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FORWARD 


Increased  concern  for  the  accountability  of  public  services  and  the  efficiency 
and  effectiveness  of  transit  system  performance  has  generated  significant 
interest  throughout  the  United  States  in  the  development  and  application  of 
performance  evaluation  methodologies.     UMTA  has  funded  through  its  Section  3 
Technical  Studies  Program,  local  studies  to  facilitate  improvements  in  transit 
perforaance  monitoring  and  evaluation  practices.     The  purpose  of  these  studies 
is  to  aid  the  providers  and  supporters  of  public  transit  services  to  improve 
the  efficiency  and  effectiveness  of  transit  in  the  United  States. 

This  document  summarizes  the  process  through  which  an  evaluation  methodology 
was  developed  for  the  mid-size  transit  systems  in  Michigan  and  the  lessons 
learned  during  the  development  period.    This  report  is  important  because  it 
identifies  the  types  of  issues  that  may  confront  other  areas  that  may  be 
considering  a  similar  effort. 

Additional  copies  of  this  report  are  available  from  the  National  Technical 
Information  Service  (NTIS),  Springfield,  Virginia  22161. 
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EXECUTIVE  SUMMARY 


Since  the  early  lOGO's,   public  involvement  in  the 
provision  of  transportation  services  has  dramatically 
increased.     Once  operated  exclusively  by  private  enterprise, 
transportation  is  now  often  viewed  as  a  service  which 
provides  benefits  to  the  community  and  therefore  merits 
public  support.      Interest  in  the  provision  of  transportation 
services  has  resulted  in  significant  increases  in  public 
spending  for  facilities,  equipment  and  operation.  Accom- 
panying the  increased  spending  has  been  an  emerging  concern 
for  the  effectiveness  and  efficiency  of  the  management  and 
operation  of  publicly  supported  transportation  services. 

State  governments  and  other  agencies  that  fund  trans- 
portation services  are  becoming  more  routinely  involved  in 
their  evaluation.     The  increased  support  for  transportation 
finance  by  state  and  local  governments  will  lead  to  increased 
concern  for  accountability  and  the  efficiency  and  effective- 
ness of  these  services. 

States  such  as  Pennsylvania,  California,  New  York,  and, 
more  recently,  Michigan  have  led  the  nation  in  implementing 
requirements  for  statewid'e  reporting  on  public  transporta- 
tion.    Other  states  or  organizations  now  considering  the 
development  and  implementation  of  similar  evaluation  pro- 
cedures can  benefit  from  the  experience  of  these  states. 
Familiarity  with  their  success  and  problems  can  facilitate 
development  of  effective  evaluation  activities. 

The  objective  of  this  report  is  to  share  the  experience 
gained  during  the  development  of  a  performance  evaluation 
methodology  for  public  transportation  in  the  State  of 
Michigan.     The  report  documents  the  process  through  which  an 
evaluation  methodology  was  developed  including  a  review  of 
project  objectives,  milestones,  meetings  and  products.  The 
report  is  not  intended  to  provide  a  detailed  review  or  tech- 
nical assessment  of  the  methodology  itself.     Although  the 
paper  is  based  on  the  experience  of  one  state  government 
agency,  the  findings  and  conclusions  are  generally  appli- 
cable to  other  organizations  that  may  evaluate  public 
transportation  performance,  particularly  those  that  provide 
or  administer  funding. 


OVERVIEW  OF  THE  DEVELOPMENT  AND  TESTING  OF  A  TRANSIT 
EVALUATION  PROGRAM  IN  MICHIGAN 

Several  bills  and  acts  passed  by  the  Michigan  State 
Legislature  during  the  fall  of  1978  called  for  improved 
information  from  state  agencies  to  demonstrate  the  effi- 
ciency and  effectiveness  of  state  funded  transportation 
programs.  Specifically: 


o    Act  No.  51,  Section  lOH(l)(b)  requires  the  State 
Transportation  Commission  to  prepare  a  progress 
report  to  the  state  legislature,  Department  of 
Management  and  Budget  (DMB)  and  Auditor  General  each 
year  by  April  1  accounting  for  the  use  of  state 
funds  for  transportation  during  the  previous  year; 
and 

o    Act  468,  an  appropriation  bill,   requires  each 

department  or  agency  to  furnish  resource  and  perfor- 
mance data  to  demonstrate  the  efficiency  and  effec- 
tiveness of  each  program  or  project  and  if  necessary 
to  prepare  a  report  explaining  why  anticipated 
levels  of  productivity  cannot  be  met. 


Following  the  passage  of  this  legislation,   the  Bureau 
of  Urban  and  Public  Transportation  (UPTRAN)  of  the  Michigan 
Department  of  Transportation  initiated  a  project  to  Develop 
and  Test  an  Evaluation  Methodology  for  the  Mid-size  Transit 
Systems  in  Michigan.     The  development  of  this  methodology 
was  intended  to  serve  as  the  first  step  toward  the  develop- 
ment and  application  of  evaluation  methodologies  for  each 
transportation  mode  receiving  financial  assistance  from  the 
State  of  Michigan. 

The  project,   initiated  by  UPTRAN  in  1978,   included  four 
phases  and  was  conducted  between  February  and  October  1979. 

o    Phase  I  included  research  on  prior  experience  in 
transit  system  evaluation  methodologies. 

o    Phase  II  involved  developing  an  evaluation  method- 
ology based  on  Phase  I  research  and  the  requirements 
for  evaluation  established  by  the  Michigan  legisla- 
ture and  UPTRAN. 

o    Phase  III  included  a  pilot  application  of  the 

methodology.     (The  pilot  test  was  based  on  data  from 

one  transit  system  in  Michigan  and  from  transit 

systems  outside  of  Michigan.) 

o    Phase  IV  involved  documenting  the  methodology  and 
pilot  application  results  as  well  as  producing  a 
manual  which  defined  the  data  routinely  used  in  the 
evaluations . 


The  project  was  conducted  by  UPTRAN  in  conjunction  with  a 
project  steering  committee.     The  committee  met  at  the 
initiation  of  each  phase  and  at  interim  project  milestones. 
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Following  the  completion  of  the  project  in  which  the 
evaluation  methodology  was  developed  and  pilot  tested, 
UPTRAN  intended  to  implement  the  evaluation  program.  This 
effort  met  with  resistance  from  representatives  of  the 
mid-size  transit  operators  in  Michigan  who  insisted  that  the 
methodology  should  first  be  more  extensively  tested,  through 
pilot  application  in  each  of  Michigan's  ten  mid-size  transit 
systems  and  then  subsequently  refined  by  incorporating 
transit  operator  input. 

In  response  to  the  operators'   concerns,   UPTRAN  initiated 
a  second  project  to  undertake  more  extensive  testing  and 
refinement  of  the  evaluation  methodology.     This  project  is 
currently  in  progress  and  should  be  completed  by  December 
1981. 

The  primary  objective  of  this  report  is  to  share  the 
experience  gained  in  Michigan  with  other  states  and  local 
areas  that  may  be  initiating  similar  efforts  to  evaluate 
public  transportation  services.     The  detailed  account  of  the 
Michigan  experience  also  sets  forth  the  context  in  which 
this  experience  was  gained. 

LESSONS  LEARNED   IN  THE  DEVELOPMENT  OF  TPIE  EVALUATION 
METHODOLOGY 

Two  major  lessons  were  learned  during  the  development 
of  the  evaluation  methodology  for  mid-size  transit  systems 
in  Michigan.     First,   there  is  a  need  to  establish  the 
groundwork  for  conducting  performance  evaluations.  Second, 
the  concerns  and  perspectives  of  public  transportation 
systems  regarding  performance  evaluation  must  be  addressed 
and  integrated  into  the  development  and  implementation  of 
the  evaluation  program. 

Importance  of  Establishing  the  Groundwork  for  Conducting 
Evaluations  of  Public  Transportation  Performance 

It  is  important  that  a  state  or  agency  interested  in 
evaluating  public  transportation  performance  establish  the 
groundwork  for  conducting  evaluations.     Such  efforts  will 
improve  the  state  or  funding  agency's  effectiveness  by: 
1)  enhancing  its  ability  to  develop  an  evaluation  program, 
and  2)  facilitating  the  timely  implementation  of  evaluation 
procedures.     The  issues  that  should  be  addressed  and  activi- 
ties that  should  be  performed  to  establish  the  groundwork 
for  conducting  the  evaluations  include: 

o    establishing  the  basis  of  authority  to  conduct 
performance  evaluations; 
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o    defining  the  objectives  for  the  evaluation  and 

specifying  the  intended  uses  of  evaluation  results; 

o    developing  an  evaluation  methodology  and  evaluation 
process ; 

o    preparing  for  the  implementation  of  the  evaluation 
program;  and 

o    communicating  the  above  information  to  the  public 
transportation  systems. 


A  detailed  discussion  of  these  activities  follows. 

It  is  critical  for  the  agency  initiating  the  evaluation 
program  to  first  establish  its  basis  of  authority  to  evalu- 
ate performance  and  define  the  evaluation  objectives  and 
intended  uses  of  evaluation  results  before  an  evaluation 
methodology  is  developed.     These  first  two  steps  define  the 
overall  responsibilities  of  participants  and  the  framework 
for  developing  the  evaluation  procedures. 

With  the  initial  groundwork  established,  the  evaluation 
methodology  and  the  evaluation  process  can  be  developed. 
The  evaluation  methodology  and  process  are  distinct  in  that 
the  methodology  includes  the  analytical  techniques  while  the 
process  specifies  how  the  activities  will  be  carried  out  in 
terms  of  the  timing  of  activities,   roles  of  participants, 
and  related  concerns  associated  with  an  ongoing  evaluation 
program. 

An  evaluation  methodology  will  be  mo're  readily  accepted 
if  its  analytical  techniques  are  straightforward  and  it  does 
not  impose  substantial  time  requirements  on  the  public 
transportation  systems  for  data  gathering  and  analysis. 
This  can  be  facilitated  by  making  maximum  use  of  available 
information.     Development  of  the  evaluation  process  should: 
1)  consider  the  current  and  ongoing  commitments  of  the 
systems,  since  an  overly  ambitious  schedule  is  likely  to  be 
unachievable,  and  2)  include  the  opportunity  for  operator 
review  and  rebuttal. 

Once  the  evaluation  methodology  and  process  are  devel- 
oped, a  period  of  transition  before  implementation  should  be 
initiated.     The  objectives  of  the  transition  period  are 
to: 

o     allow  for  final  testing  and  refinement  of  the 
evaluation  methodology  and  process;  and 

o    ensure  that  all  participants  are  trained  and  informed 
as  appropriate. 
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The  importance  of  this  transition  period  should  not  be 
underestimated.     A  period  of  one  year  is  a  reasonable 
benchmark  for  conducting  these  activities. 

Communication  between  the  evaluator  and  the  systems 
subject  to  evaluation  may  occur  at  each  stage  of  the  devel- 
opment,  testing,  and  ultimate  implementation  of  an  evalua- 
tion program.     However,  the  nature  and  frequency  of  communi- 
cation will  vary  from  area  to  area.     In  certain  areas,  some 
or  all  of  the  public  transportation  systems  will  be  directly 
involved  in  decision  making  to  develop  and  implement  the 
evaluation  program.     In  other  areas,   the  communication  with 
the  systems  may  be  more  informational. 

Regardless  of  the  level  of  public  transportation  system 
involvement,   lines  of  communication  should  be  established  so 
that  system  concerns  and  recommendations  are  considered  and 
requested  information  is  readily  available.     At  a  minimum, 
all  systems  that  will  be  evaluated  should  be  fully  informed 
about  the  evaluation  program  before  it  is  implemented. 

Exhibit  A  summarizes  important  considerations  for 
states  or  agencies  including  the  timing  of  each  of  these 
five  activities. 

Public  Transportation  System  Concerns  and  Perspectives 
Regarding  Performance  Evaluation 

The  concerns  and  perspectives  of  public  transportation 
systems  regarding  the  evaluation  of  their  performance  by 
states  or  other  funding  agencies  will  vary  from  system  to 
system.     In  general,  however,  the  following  questions  are 
likely  to  be  raised: 

o     How  will  the  evaluation  results  be  used? 

o    Will  the  allocation  of  funding  be  affected  by  the 
results  of  the  evaluations?     If  so,  how? 

o    What  will  the  evaluations  require  of  the  public 

transportation  systems  in  terms  of  staff  and  manage- 
ment time  and  data? 

o    Who  will  conduct  the  evaluations? 

o    What  recourse  does  a  system  have  if  it  disagrees 
with  the  evaluation  conclusions? 

o    What  benefits  can  be  expected  from  the  evaluation 
results? 
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o     Is  state  or  funding  agency  involvement  in  public 
transportation  system  management  and  operation 
exceeding  established  limits  of  authority  and 
responsibility? 

o    Are  peer  comparisons  valid? 

o    Are  the  proposed  evaluation  procedures  redundant 

with  current  reporting  or  program  monitoring  activi- 
ties for  the  system? 


Simply  stated,  a  primary  concern  of  public  transporta- 
tion systems  is  how  they  may  be  affected  by  evaluations  of 
their  performance  by  an  outside  agency.     Specifically,  the 
systems  are  concerned  about  how  the  evaluations  may  affect: 

o    financial  resources; 

o    staff  resources; 

o    control  of  management  over  internal  system  opera- 
tions and  decisionmaking;  and 

o    local  officials'  and  the  public's  perceptions  of  the 
system  and  its  management. 


Each  of  these  issues  was  raised  by  mid-size  transit  systems 
in  Michigan  and  is  discussed  throughout  this  report. 
Similar  issues  arose  in  Pennsylvania  and  California  when 
statewide  performance  evaluations  were  initiated. 

Exhibit  B  summarizes  the  concerns  of  public  transporta- 
tion systems  regarding  their  evaluation  by  an  outside  agency 
and  presents  the  suggested  response  of  the  evaluator. 

The  conclusions  of  the  Michigan  experience  with  regard 
to  the  concerns  and  perspectives  of  public  transportation 
systems  are  that: 

o     It  is  the  responsibility  of  the  evaluator  to 
promote  goodwill.     Consequently,  the  evaluation 
program  should  focus  on  service  improvement  rather 
than  penalty  and  equal  effort  should  be  given  to 
identifying  the  progress  and  strengths  of  each 
system  as  well  as  opportunities  to  improve. 

o    The  potential  effects  of  an  evaluation  program  on 
financing  should  be  determined  at  the  earliest 
possible  date  and  communicated  to  the  systems  so 
that  they  can  avoid  a  loss  of  funds. 
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IX 


o    Time  demands  of  the  evaluation  on  public  transpor- 
tation system  personnel  should  be  minimized  by 
making  use  of  available  information,  avoiding 
redundant  requirements,  and  simplifying  evaluation 
procedures . 

o    Evaluations  should  generally  not  extend  the  role  of 
the  evaluator  into  day-to-day  management  activities. 
The  authority  and  responsibilities  of  the  evaluator 
should  not  interfere  with  or  impede  the  authority 
and  responsibilities  of  public  transportation  system 
management . 


CONCLUS ION 

The  experience  gained  in  Michigan  in  the  development  of 
a  performance  evaluation  program  for  public  transportation 
is  presented  to  assist  others  to  develop  their  program  more 
effectively  and  expeditiously.     UPTRAN's  project  to  develop 
an  evaluation  methodology  for  its  mid-size  transit  systems 
lends  important  documentation  for  use  by  the  growing  number 
of  states,  other  funding  agencies,  and  others  interested  in 
considering  performance  evaluation  programs. 

The  Michigan  experience  confirms  that  the  process  for 
an  outside  agency  to  develop  and  implement  an  evaluation 
program  is  time-consuming  and  sensitive.     Two  important 
lessons  were  learned  through  the  experience  gained  in 
Michigan.     First,   it  is  important  that  the  groundwork  for 
evaluations  be  carefully  established.     The  development  of  an 
evaluation  methodology  represents  only  one  of  a  series  of 
important  activities  to  establish  this  groundwork.  Develop- 
ing the  methodology  must  be  preceded  by  establishing  1)  the 
authority  to  evaluate,   2)  the  objectives  of  the  evaluation, 
and  ?)  the  intended  uses  of  evaluation  results.     Once  devel- 
oped, the  evaluation  methodology  and  process  for  its  appli- 
cation must  be  tested  and  refined  in  conjunction  with  the 
public  transportation  systems  that  will  be  evaluated. 

The  second  important  lesson  learned  in  Michigan  is  that 
the  concerns  and  perspectives  of  the  public  transportation 
systems  subject  to  evaluation  must  be  recognized  and  con- 
sidered.    Throughout  the  development  and  implementation  of 
an  evaluation  program,  it  is  important  that  the  organization 
initiating  the  evaluations  communicate  with  the  public 
transportation  systems  and  is  responsive  to  their  concerns 
and  perspectives. 
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I.  INTRODUCTION 


The  objective  of  this  report  is  to  share  the  experience 
gained  during  the  development  of  a  performance  evaluation 
methodology  for  public  transportation  in  the  State  of 
Michigan.      Including  this   Introduction,   the  report  consists 
of  seven  sections  and  two  appendices.     Subsequent  sections 
are  described  below: 

o     Section  II  Background  -  provides  information  on  the 
Bureau  of  Urban  and  Public  Transportation  (UPTRAN) 
and  its  mandate  to  evaluate  public  transportation  in 
Michigan . 

o     Section  III   Initiation  of  the  Project  to  Develop  an 
Evaluation  Program  for  Mid-size  Transit  Systems  - 
describes  the  objectives  of  the  project,   the  selec- 
tion of  a  consultant  and  the  formation  of  the 
project  steering  committee. 

o     Section  IV  Development  and  Testing  of  the  Evaluation 
Methodology  -  provides  a  detailed  review  of  each 
phase  of  the  project,   and  the  activites  of  each 
project  steering  committee  meeting. 

o     Section  V  Project  to  Test  and  Refine  the  Evaluation 
Methodology  for  fAid-size  Transit  Systems  in  Michigan 
-  describes  the  initiation  of  the  current  project  to 
conduct  a  statewide  pilot  test  of  the  evaluation 
methodology   (i.e.,  pilot  test  in  each  of  the  transit 
systems  to  be  routinely  evaluated)   and  refine  the 
methodology  with  transit  operator  input. 

o     Section  VI  Lessons  Learned  in  the  Development  of  the 
Evaluation  Methodology  for  Mid-Size  Transit  Systems 
in  Michigan  -  first  describes  the  importance  of 
establishing  the  groundwork  for  conducting  evalua- 
tions and  presents  each  of  the  major  activities  that 
should  be  addressed.     It  then  discusses  the  key 
concerns  and  perspectives  of  public  transportation 
systems  with  regard  to  performance  evaluation  and 
suggests  the  appropriate  role  of  the  evaluator  in 
response  to  each  concern. 

o    Section  VII  Conclusion  -  briefly  highlights  the 
significance  and  key  findings  of  this  report. 

o     Appendix  A  is  a  letter  sent  by  UPTRAN  to  the  transit 
operators  and  the  steering  committee  members  sum- 
marizing the  results  of  the  final  meeting  for  the 
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Project  to  Develop  and  Test  an  Evaluation  Method- 
ology for  Mid-Size  Transit  in  Michigan. 

o    Appendix  B  is  an  UPTRAN  statement  on  Public  Trans- 
portation System  Evaluation. 
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II.  BACKGROUND 


This  section  of  the  report  contains  information  on 
UPTRAN's  program  responsibilities  and  its  mandate  to  evalu- 
ate public  transportation.      It  provides  a  background  for 
understanding  the  projects  conducted  by  UPTRAN  to  initially 
develop  and  test  and  subsequently  to  further  test  and  refine 
an  evaluation  methodology  for  public  transportation  in 
Michigan . 


UPTRAN  PROGRAM  RESPONSIBILITIES 

UPTRAN  is  responsible  for  the  annual  administration  of 
the  State  Comprehensive  Transportation  Fund  for  the  Michigan 
Public  Transportation  Program.     The  fund  provides  capital 
and  operating  grants,   loans,  and  demonstration  grants  to  the 
following  seven  public  transportation  modes  in  Michigan: 

o    Local  transit; 

o     Interim  Elderly  and  Handicapped  and  16(b)(2) 
Transit;!/ 

o  Rail  passenger; 

o  Rail  freight; 

o  Intercity  bus; 

o  Water  transportation;  and 

o  Air  commuter. 


The  evaluation  program  discussed  in  this  report  was 
developed  for  the  local  transit  systems  serving  urbanized 
areas  in  Michigan  with  a  population  of  at  least  50,000, 
excluding  the  Southeastern  Michigan  Transportation  Authority 
(SEMTA).     Currently  there  are  ten  transit  systems  in  Michi- 
gan in  this  group:     Ann  Arbor,  Battle  Creek,   Bay  County, 


_!/     This  program  provides  vehicles  and  operating  grants  to 
local  units  of  government  in  areas  where  public  trans- 
portation is  non-existent.      In  addition,   the  state  pays 
the  20  percent  matching  share  for  all  vehicles  purchased 
through  the  federal  16(b)(2)  program  for  senior  citizens 
and  the  transportation  handicapped  and  limited  assis- 
tance to  operate  these  vehicles. 
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Flint,  Grand  Rapids,  Jackson,  Kalamazoo,  Lansing,  Muskegon, 
and  Saginaw. 

Operating  and  capital  assistance  are  provided  by  the 
state  to  all  eligible  Michigan  bus  systems  operating  in 
urbanized  areas.     Urbanized  area  bus  systems  other  than 
SEMTA  are  allocated  up  to  one  third  of  their  eligible 
operating  expenses  by  formula.     (SEMTA  receives  up  to  21 
percent  of  its  operating  expenses  through  state  assistance.) 
Up  to  one  hundred  percent  of  the  non-federal  share  of 
capital  expenses  (i.e.,  20  percent  of  the  total  capital 
expenses)  are  paid  through  state  grants. 


UPTRAN's  MANDATE  TO  EVALUATE  PUBLIC  TRANSPORTATION 

Several  bills  and  acts  passed  by  the  Michigan  State 
Legislature  during  the  fall  of  1978  called  for  improved 
information  from  state  agencies  to  demonstrate  the  effi- 
ciency and  effectiveness  of  state  funded  transportation 
programs.  Specifically: 

o    Act  No.  51,  Section  10H(l)(b)  requires  the  State 
Transportation  Commission  to  prepare  a  progress 
report  to  the  state  legislature.  Department  of 
Management  and  Budget  (DMB)  and  Auditor  General  each 
year  by  April  1  accounting  for  the  use  of  state 
funds  for  transportation  during  the  previous  year; 
and 

o    Act  468,  an  appropriation  bill,  requires  each 

department  or  agency  to  furnish  resource  and  perfor- 
mance data  to  demonstrate  the  efficiency  and  effec- 
tiveness of  each  program  or  project  and  if  necessary 
to  prepare  a  report  explaining  why  anticipated 
levels  of  productivity  cannot  be  met. 


UPTRAN  interpreted  the  legislation  passed  in  1978  as  a 
mandate  to  1 )  develop  evaluation  methodologies;   2)  perform 
evaluations  of  the  programs  which  receive  funds  from  the 
State  Comprehensive  Transportation  Fund;  and  3)  report 
evaluation  findings  for  each  of  these  public  transportation 
programs.     Acting  on  its  interpretation  of  the  1978  legisla- 
tion, UPTRAN  planned  to  develop  evaluation  methodologies  for 
use  in  all  public  transportation  modes  funded  by  the  State 
Comprehensive  Transportation  Fund  and  to  annually  apply  the 
methodologies  and  use  the  results  of  the  evaluation  as  the 
basis  for  the  annual  report  required  by  Act  51. 

It  is  important  to  note  that  while  the  1978  legislation 
indicated  that  the  state  should  have  information  to  ensure 
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accountability  and  to  report  on  the  efficiency,  effective- 
ness and  progress  of  state  funded  transportation  programs, 
it  did  not  explicitly  stipulate  that  an  evaluation  of  the 
public  transportation  programs  by  state  agencies  is  required. 
It  was  UPTRAN's  interpretation  of  the  legislative  acts  that 
the  state  agencies  must  develop  evaluation  methodologies 
and,  on  an  annual  basis,   evaluate  the  state  funded  public 
transportation  programs  to  produce  the  information  requested 
by  the  legislation.     Representatives  of  the  mid-size  transit 
systems  did  not  concur  with  this  interpretation.  They 
believed,  in  fact,   that  there  was  not  a  firm  agreement  among 
the  legislators  themselves  on  a  clear  interpretation  of  the 
legislation. 

In  May  1979,  however,  the  annual  report  of  UPTRATI 
activities  prepared  by  the  State  Auditor  General  supported 
UPTRAN's  interpretation  of  the  state  legislation.  The 
report  noted  that  the  bureau  (UPTRAN)  had  not  developed 
criteria  and  methods  to  evaluate  the  performance  and  effec- 
tiveness of  bus  transportation  systems.     The  report  recom- 
mended that  the  Department:     1)  develop  criteria  and  methods 
to  determine  if  state  gas  taxes  are  used  to  operate  efficient 
and  effective  public  transportation  systems;   and  2)  in 
cooperation  with  the  agencies  and  authorities  develop 
uniform  policies  and  methods  to  manage  and  control  the 
operating  costs  of  bus  transportation  systems  and  routes. 

The  May  1979  State  Auditor's  report,  a  statement  by 
Governor  Milliken  outlining  the  Governor's  Budget  Develop- 
ment Policy  for  1980-8li/  and  the  state  legislation  passed 
in  1978,  have  regularly  been  cited  by  UPTRAN  as  the  basis  of 
UPTRAN's  mandate  to  develop  evaluation  methodologies, 
routinely  evaluate  public  transportation  program  perfor- 
mance, and  annually  report  to  the  State  Legislature,  DMB  and 
the  State  Auditor  General  on  the  efficiency,  effectiveness 
and  progress  of  the  state-funded  public  transportation 
programs . 


1^1     The  Governor's  Budget  Development  Policy  Statement  for 

1980-81  called  for  greater  accountability,  emphasized  program 
efficiency  and  effectiveness,  and  stressed  the  importance  of 
improved  information  to  facilitate  enlightened  decision- 
making by  the  Michigan  State  Legislature  and  citizens. 
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III.      INITIATION  OF  THE  PROJECT  TO  DEVELOP  AN 

EVALUATION  PROGRAM  FOR  MID-SIZE  TRANSIT  SYSTEMS 


In  the  fall  of  1978  UPTRAN  initiated  a  project  to 
develop  an  evaluation  program  for  the  mid-sizel./  local 
transit  systems  in  Michigan  which  receive  formula  operating 
assistance  for  up  to  one-third  of  their  eligible  operating 
expenses.     This  project  was  to  be  the  first  step  in  carrying 
out  UPTRAN 's  plan  to  develop  methodologies  and  annually 
evaluate  each  of  the  public  transportation  modes  that  are 
supported  by  the  State  Comprehensive  Transportation  Fund. 

The  objectives  of  the  UPTRAN  project  were  to  develop 
and  test  an  evaluation  methodology  for  the  mid-size  local 
transit  systems,  and  train  UPTRAN  staff  in  its  application 
and  use.     This  was  to  be  accomplished  in  one  year  (largely 
the  1979  calendar  year),   with  the  intent  that  UPTRAN  would 
apply  the  methodology  "live"  in  the  following  year. 

During  the  inception  of  the  project,   there  was  consid- 
erable enthusiasm  within  UPTRAN  regarding  potential  appli- 
cations of  the  evaluation  results  both  by  the  state  and 
local  governments.     In  particular,  although  it  was  not  an 
objective  of  the  project,   the  Research  and  Evaluation 
Section  of  UPTRAN  (which  conducted  the  project)  assumed  that 
by  April  1,   1980,   the  annual  report  required  by  Act  51  would 
be  based,  in  part,  on  the  results  of  the  first  state-wide 
evaluation  of  Michigan  mid-size  transit  systems. 

The  Research  and  Evaluation  Section  of  UPTRAN,  which 
prepares  the  annual  report,   viewed  the  future  availability 
of  the  annus^l  evaluation  results  as  an  opportunity  to 
improve  the  quality  and  usefulness  of  the  reports  and  the 
responsiveness  of  UPTRAN  to  Act  51.     By  including  the 
evaluation  results,   the  reports  could  more  thoroughly 
address  the  efficiency,  effectiveness  and  progress  of  the 
transportation  programs. 


SELECTION  OF  A  CONSULTANT 

Through  a  competitive  procurement,  which  included  the 
preparation  of  a  written  proposal  and  an  oral  presentation, 


1_/     These  bus  systems  are  commonly  referred  to  as  the 

mid-size  systems  since  they  are  smaller  than  SEMTA  and 
larger  than  the  smaller  urban  and  rural  systems. 
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a  contractor  was  selected  to  assist  UPTRAN  to  develop  and 
test  an  evaluation  methodology  for  the  mid-size  transit 
systems  in  Michigan. 

The  request  for  proposal  (RFP),   dated  September  22, 
1978,  stated  that  the  objectives  of  the  study  were  to: 

o    develop  a  methodology  for  evaluating  the  performance 
of  urban  transit  systems  in  Michigan  based  on  joint 
state-local  perspectives; 

o     train  department  staff  in  the  application  of  the 
methodology  and  in  disseminating  the  methodology  to 
other  transportation  evaluation  staffs  in  Michigan; 
and 

o    apply  the  methodology  in  a  pilot  evaluation  of  a 
public  transportation  system  to  be  selected  jointly 
by  the  Bus  Transport  Division  of  the  Bureau  of 
Public  Transportation  and  the  Michigan  Public 
Transit  Association. 


The  RFP  also  stated  that  application  of  the  methodology 
should  annually  provide  information  useful  to  the  state, 
transit  systems  management  and  elected  officials  in  the 
area.     To  assure  that  state  and  local  objectives  would 
be  met,   the  RFP  specified  that  the  methodology  would  be 
developed  in  consultation  with  Department  (UPTRAN)  staff, 
local  government  officials  from  the  pilot  area  and  the  MPTA. 

Exhibit  1  presents  an  overview  of  the  selected  contrac- 
tor's proposed  work  plan  and  schedule. 


FORMATION  OF  THE  PROJECT  STEERING  COMMITTEE 

As  part  of  the  project,   UPTRAN  suggested  the  composi- 
tion of  a  project  steering  committee  and  suggested  that 
formation  of  the  committee  be  included  in  Phase  I  rather 
than  Phase  1 1  of  the  project  as  initially  proposed.  This 
was  the  only  change  to  the  proposed  work  plan  requested  by 
UPTRAN . 

The  steering  committee  included  representatives  from: 
o     the  UPTRAN  Bus  Transport  Division; 
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EXHIBIT  1 

DEVELOP  AND  TEST  AN  EVALUATION  METHODOLOGY 
FOR  MID-SIZE  TRANSIT  SYSTEMS  IN  MICHIGAN 
PROPOSED  PROJECT  SCHEDULE 


TASK  AND  SUBTASK 

ONE 

MONTH 
TWO 

S  FROM  Pfl 
THREE 

OJECT  IN 
FOUR 

TIATION 
FIVE 

SIX 

PHASE  1— RESEARCH  AND  REPORT  ON  TRANSIT  SYSTEM 
EVALUATION  METHODOLOGIES 

Task  1  Prepare  and  Submit  Detailed  Work  Plan 

Task  2  Survey  Transit  System  Evaluation  Methodologies 

Task  3  Reoort  on  Transit  Systems  That  Have  Conducted  Evaluations 

3.1  Document  Phase  1  Findings 

3.2  Client  Meeting  to  Review  Phase  1  and  Initiate  Phase  II 
PHASE  II— DEVELOP  EVALUATION  METHODOLOGY 

Task  4  Develop  an  Evaluation  Methodology  for  Mid-Sized  Michigan  Communities 

4.1  Organize  Steenng  Committee 

4.2  Establish  Goals  ano  Ob;ecti'/es 

4.3  Identify  Preliminary  Performanca  Indicators  and  Suggest  Standards 

4.4  Determine  Data  Needs  and  Prepare  Data  Manual 

4.5  Develop  Data  Analysis  and  Evaluation  Techniques 

4.6  Document  Evaluation  Methodology 

4.7  Conduct  Steenng  Committee  Work  Shop 

PHASE  III— CONDUCT  PILOT  PERFORMANCE  EVALUATION 

Task  5  Apply  Evaluation  Methodology  in  Pilot  Performance  Evaluation 

5.1  Select  Area  for  Pilot  Evaluation 

5.2  Conduct  Interviews  witti  Management 

5.3  Taikjr  Evaluation  Methodology 

5.4  Analyze  Data  and  Conduct  Overall  Evaluation 

5.5  Conduct  Detailed  Investigation  As  Appropnate 

Task  6  Document  Pilot  Evaluation  Results 

6.1  Document  Evaluation  Findings 

6.2  Review  Results  with  Candidate  System 

6.3  Review  Results  with  Steenng  Committee  and  Initiate  Phase  IV  Activities 

PHASE  IV— PREPARE  TRANSIT  EVALUATION  MANUAL 

Task  7  Prepare  Transit  Evaluation  Manual  and  Tram  Staff 

7.1  Prepare  Draft  Manual  and  Training  Materials 

7.2  Revise  Draft  Manual  and  Training  Materials  ' 

7.3  Conduct  Training  Session 

 1 

i 

 A 

▲ 

 <\ 

▲ 

'  -J 

scrwduMd  sroiect  me«(ings 
□    aratt  linal  reoon 
■     final  reoort 
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^  intanm  oraiect  reoorts 


8 


o     the  GEMTA  Budget  Division;!/ 

o     the  Michigan  Public  Transit  Association  (MPTA) ; 

o     the  MDOT  Interagency  Transportation  Coordinating 
Committee ; 

o    an  advocate  for  the  transportation  handicapped; 

o     the  Southeast  Michigan  Council  of  Governments 
(SEMCOG):  1/ 

o     the  City  of  Lansing  Planning  Department; 
o    the  MDOT  Planning  Department; 

o     the  Tri-County  Regional  Planning  Commission;  and 
o     the  Michigan  Department  of  Management  and  Budget. 


The  representative  of  the  MPTA  on  the  project  steering 
committee  served  to  represent  all  of  the  mid-size  transit 
systems.     UPTRAN  had  originally  contacted  the  Capital  Area 
Transportation  Authority  (CATA)  requesting  its  participation 
on  the  committee.     The  invitation  was  declined  and  UPTRAN 
was  informed  that  the  MPTA  could  serve  as  representative  for 
the  transit  systems. 


_!/     A  representative  from  the  SEMTA  Budget  Division  and 

SEMCOG  were  included  in  the  project  steering  committee 
even  though  they  are  not  from  an  area  in  Michigan  with  a 
mid-size  transit  system  because  of  their  expertise  and 
interest  in  transit  financial  reporting  and  transit 
analysis  and  planning,  respectively. 
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IV.     DEVELOPMENT  AND  TESTING  OF 
THE  EVALUATION  METHODOLOGY 


The  project  to  develop  and  test  an  evaluation  method- 
ology for  mid-size  transit  systems  in  Michigan  was  conducted 
in  four  phases. 

o    Phase  I  included  research  of  prior  experience  of 
transit  system  evaluation  methodologies. 

o    Phase  II  involved  developing  an  evaluation  methodol- 
ogy based  on  Phase  I  research  and  the  requirements 
for  evaluation  in  Michigan  established  by  the 
Michigan  legislature  and  UPTRAN. 

o    Phase  III  included  a  pilot  application  of  the 
methodology. 

o    Phase  IV  involved  documenting  the  methodology  and 
pilot  application  results  as  well  as  producing  a 
manual  which  defined  the  data  routinely  used  in  the 
evaluations . 


The  activities  in  each  phase  including  product  develop- 
ment and  the  outcome  of  the  project  steering  committee 
meetings  are  discussed  below. 

PHASE   I  -  RESEARCH  AND  REPORT  ON  TRANSIT  SYSTEM  EVALUATION 
METHODOLOGIES 

The  first  phase,  initiated  on  February  15,   1979  and 
conducted  during  the  first  six  weeks  of  the  project,  involved 
a  state  of  the  art  review  of  methodologies  used  to  evaluate 
transit  systems.     A  report  documenting  Phase  I  results  was 
prepared  and  distributed  to  the  project  steering  committee 
members  prior  to  the  first  steering  committee  meeting. 

The  first  section  of  the  report  discussed  (1)  why 
transit  performance  evaluations  are  conducted,    (2)  common 
objectives  of  transit  evaluation  results,  and  (3)  the 
situation  in  Michigan  that  intitiated  interest  in  perfor- 
mance evaluation.     The  discussion  of  the  Michigan  situation 
largely  reviewed  the  legislation  passed  in  1978  by  the 
Michigan  State  Legislature. 

The  second  section  of  the  report  discussed  current 
experience  with  transit  performance  evaluation.     The  last 
two  sections  of  the  report  outlined  a  process  for  developing 
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a  performance  evaluation  methodology  and  presented  a  prelim- 
inary framework  for  public  transportation  performance 
evaluation  in  the  State  of  Michigan. 

First  Project  Steering  Committee  Meeting 

The  first  project  steering  committee  meeting  was  held 
in  early  April,   1979.     The  objectives  of  the  meeting  were 
to: 

o     review  the  objecives,   approach,   and  schedule  for  the 
study ; 

o    review  Phase  I  findings; 

o    establish  goals  and  objectives  for  the  evaluation 
methodology  for  the  mid-size  transit  systems  in 
Michigan; 

o    discuss  a  suggested  evaluation  approach; 

o    review  recommended  performance  indicators; 

o    discuss  a  data  needs  manual  (which  would  be  used  to 
develop  performance  indicators); 

o     introduce  suggested  data  analysis  techniques;  and 

o     identify  candidate  transit  systems  for  the  pilot 
evaluation . 


During  the  meeting  the  discussion  focused  largely  on 
the  use  of  the  evaluation  results,  particularly  their 
potential  use  in  the  allocation  of  financial  resources,  and 
the  responsibilities  of  UPTRAN  in  relation  to  the  public 
transportation  systems. 

More  specifically,  the  major  issues  raised  by  the 
steering  committee  members  included: 

o    a  discussion  of  whether  UPTRAN  intended  to  develop 
standards  for  performance  which  might  conflict  with 
local  goals  and  objectives; 

o  a  clarification  of  UPTRAN's  role  regarding  whether 
it  was  to  serve  as  a  pass  through  agency,  distrib- 
uting earmarked  funds,  or  as  monitor  or  watch  dog; 
and 
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o    an  objection  that  was  raised  to  comparing  public 
transportation  systems  as  part  of  the  evaluation. 


In  response  to  the  first  issue,  UPTRAN  stated  that  it 
would  not  mandate  the  achievement  of  prescribed  standards 
for  public  transportation  performance  in  the  state. 

In  response  to  the  second  issue,  UPTRAN 's  authority  to 
evaluate  program  performance  was  discussed  and  acknowledged. 
UPTRAN  stated  that  while  operating  assistance  to  local  bus 
systems  in  Michigan  is  allocated  by  formula  (established  by 
the  state  legislature  and  therefore  passed  through  UPTRAN  to 
the  transit  systems),  UPTRAN  has  the  authority  to  assess  and 
report  on  the  efficiency,  effectiveness,  and  progress  of  the 
recipients  of  these  funds.     In  addition,  UPTRAN  is  responsi- 
ble for  the  allocation  of  state  funds,  other  than  operating 
assistance,  including  capital  and  demonstration  grants. 
UPTRAN  stated  that  it  believed  it  was  important  to  use  all 
available  information  to  prioritize  these  grants  within  and 
among  the  eligible  systems. 

The  third  issue  was  raised  by  the  representative  of  the 
MPTA  who  stated  that  the  transit  systems  in  Michigan  are 
each  unique  and  therefore  cannot  be  meaningfully  compared. 
In  response  to  this  comment,   there  was  a  discussion  about 
the  usefulness  of  comparisons  as  a  diagnostic  tool  for 
information  gathering.     It  was  generally  agreed  that  while 
peer  comparisons  must  be  carefully  conducted,  recognizing 
the  differences  among  transit  systems,   they  can  produce 
information  which  is  informative  and  useful  in  a  preliminary 
or  diagnostic  analysis  of  performance.     This  discussion  was 
significant  since  the  proposed  evaluation  approach  includes 
a  diagnostic  phase  which  utilizes  both  peer  comparisons  and 
time  series  analysis  of  performance  indicators  to  identify 
selected  areas  for  more  detailed  review  in  each  transit 
system. 

Overall  the  steering  committee  agreed  with  the  initial 
concepts  for  the  evaluation  approach  and  the  recommended  use 
of  performance  indicators.     The  concerns  about  the  study  did 
not  address  the  technical  merit  of  the  suggested  evaluation 
approach  or  the  necessity  for  public  accountability. 
Rather,  the  concerns  focused  on  the  potential  increased 
involvement  of  the  state  in  the  provision  of  local  transit 
services,   the  threat  of  reduced  funding  or  changes  to 
funding  allocation  policies  and  the  potential  misrepresenta- 
tion of  transit  systems  as  a  result  of  comparing  system 
performance. 
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PHASE   II  -  DEVELOP  EVALUATION  METHODOLOGY 


The  second  phase  of  the  study  involved  development  of 
the  evaluation  procedures  and  the  preparation  of  a  draft 
Data  Needs  Manual,     Development  of  the  evaluation  procedures 
involved  refining  the  approach  and  analytical  techniques 
outlined  in  the  first  project  steering  committee  meeting  and 
making  the  procedures  more  detailed. 

The  draft  evaluation  methodology  developed  for  the 
mid-size  transit  systems  in  Michigan  included  a  two-phased 
procedure  for  evaluating  each  of  the  transit  systems. 
Exhibit  2  provides  an  overview  of  the  procedure  as  developed 
in  the  second  phase  of  the  study. 

The  first  phase  of  the  methodology  involves  a  diagnostic 
review  of  a  set  of  performance  indicators  which  measure 
various  aspects  of  transit  system  efficiency  and  effective- 
ness.    The  indicators  from  each  mid-size  transit  system  in 
Michigan  are  compared  across  transit  systems  and  over  time. 
Indicators  with  values  that  are  significantly  higher  or 
lower  than  those  in  the  other  systems  and  those  that  are 
changing  significant  overtime  are  identified.     These  indica- 
tors are  then  reviewed  in  preparation  for  the  second,  more 
detailed,  phase  of  the  evaluation. 

The  primary  objectives  of  the  first  phase  of  the 
evaluation  are  to  conduct  a  preliminary  review  of  each 
System  and  to  limit  the  scope  of  the  more  detailed  reviews. 
By  focusing  on  the  attributes  of  each  transit  system  that 
suggest  the  need  for  more  in-depth  analysis,   the  scope  of 
the  evaluations  and  resources  required  to  conduct  them  can 
be  limited. 

The  second  phase  of  the  evaluation  procedure  is  more 
investigative.     Through  direct  contact  with  the  transit 
operators,   information  is  gathered  about  the  local  operating 
environment  of  the  transit  system,  the  effects  of  state  and 
local  policy  on  performance,  and  the  management  and  oper- 
ating procedures  within  the  transit  system.  Information 
gathering  is  focused  on  attributes  of  the  transit  system 
suggested  by  the  indicators  identified  in  the  diagnostic 
phase  of  the  evaluation.     The  information  gathered  in  this 
second  phase  of  the  evaluation  is  intended  to: 

o    provide  information  and  explain  performance  charac- 
teristics ; 

o     identify  examples  of  innovative  or  exemplary 
performance  that  may  be  shared  among  the  local 
transit  systems  in  Michigan; 
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o    suggest  the  need  for  further  study;  and 

o     identify  opportunities  for  improvement  and  poten- 
tially recommend  solutions. 

This  information  serves  as  the  basis  for  monitoring  perfor- 
mance overtime  and  annually  assessing  the  efficiency, 
effectiveness  and  progress  of  each  transit  system. 

A  Data  Needs  Manual  was  developed  in  conjunction  with 
the  evaluation  procedures.     The  Data  Meeds  Manual  defines 
each  performance  indicator  included  in  the  diagnostic  phase 
of  the  methodology  and  explains  how  each  indicator  is 
developed.     With  the  exception  of  several  indicators  which 
include  demographic  data  (available  from  UPTRAN) ,  the 
performance  indicators  are  composed  from  financial  and 
operating  statistics  included  in  the  (required  level) 
Section  15  reporting  requirements  developed  in  the  Urban 
Mass  Transportation  Administration  (UMTA)  of  the  U.S. 
Department  of  Transportation.     Section  15  data  is  routinely 
reported  by  the  transit  operators  to  the  federal  government 
each  fiscal  year.     A  preliminary  draft  of  the  Data  Needs 
Manual  was  distributed  to  the  Steering  Committee  Members  in 
advance  of  the  second  project  Steering  Committee  Meeting. 

Second  Project  Steering  Committee  Meeting 

The  second  project  Steering  Committee  meeting  was  held 
on  April  25,    1979.     The  objectives  of  this  meeting  were 
to: 

o     review  the  evaluation  approach; 
o     present  the  Data  Needs  Manual; 

o     discuss  the  diagnostic  element  of  the  evaluation 
methodology; 

o    discuss  the  detailed  evaluation  element  of  the 
evaluation  methodology;  and 

o    discuss  the  pilot  upcoming  evaluation. 

During  the  meeting,  the  format  of  the  Data  Needs  Manual  and 
the  performance  indicators  included  in  the  methodology  were 
presented  and  discussed  in  detail.  The  presentation  includ 
ed  an  explanation  of  the  logic  behind  including  each  indica 
tor  and  the  structure  within  which  the  indicators  are  to  be 
used  in  the  diagnostic  phase  of  the  methodology. 
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The  framework  for  both  phases  of  the  evaluation  method- 
ology were  presented  first  in  overview  and  then  in  detail. 
The  merits  of  the  methodology  were  discussed  including: 

o     the  methodology  does  not  impose  excessive  reporting 
burden  on  transit  operators; 

o  a  staged  (two  phase)  evaluation  process  is  cost 
effective  because  it  focuses  the  evaluation  and 
limits  the  scope  of  analysis; 

o     the  state  gains  a  greater  understanding  of  the 

operating  environment  and  policies  of  the  raid-size 
transit  systems  in  Michigan;  and 

o    the  methodology  focuses  on  improvements  in  public 
transportation  system  efficiency  and  effectiveness. 


This  meeting  also  included  a  preliminary  discussion 
about  the  pilot  test  of  the  performance  evaluation  methodol- 
ogy.    Suggested  objectives  for  the  pilot  test  discussed  in 
this  meeting  included: 

o     testing  alternative  analytical  methods  for  conduct- 
ing the  diagnostic  phase; 

o    adding  more  detail  to  the  procedures  for  the  second 
phase  of  the  evaluation  methodology;  and 

o     refining  the  methodology  based  on  information  gained 
through  the  pilot  test. 


One  of  the  mid-size  transit  systems  in  Michigan  volunteered 
to  paT'ticipate  in  the  pilot  test.     Coincidentally ,  this  was 
the  only  transit  system  in  Michigan  to  have  completed  its 
first  annual  UMTA  Section  15  report  at  that  time. 

It  was  decided  at  the  same  time  that  the  pilot  test 
would  include  data  to  allow  both  time  series  and  peer 
comparison  analysis.     The  time  series  analysis  would  be 
performed  on  the  data  from  the  Michigan  transit  system  which 
had  volunteered  for  the  pilot  test.     The  data  used  in  this 
analysis  would  be  a  combination  of  the  first  annual  Section 
15  report   (1978  data),  quarterly  statistics  historically 
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required  by  UPTRAN.i/  and  other  data  from  the  transit 
system.     The  peer  comparison  would  be  based  on  Section  15 
data  from  the  Michigan  Transit  System  that  volunteered  for 
the  pilot  test  together  with  Section  15  data  provided  by 
UMTA  for  13  other  transit  systems  with  less  than  100  revenue 
vehicles . 

Unlike  the  first  Steering  Committee  meeting  in  which 
policy  issues  were  debated,  including  UPTRAN's  authority  to 
evaluate  the  transit  systems  and  the  use  of  evaluation 
results,   this  meeting  did  not  include  such  discussion.  The 
Steering  Committee  Members  were  supportive  of  the  informa- 
tion presented  and  did  not  request  that  the  study  be  modi- 
fied in  scope  or  focus. 


PHASE   III  -  CONDUCT  PILOT  PERFORMANCE  EVALUATION 

This  phase  of  the  study  involved  performing  the  final 
preparation  for  and  conducting  the  pilot  test  of  the  evalua- 
tion methodology  and  reviewing  and  documenting  the  pilot 
test  results.     Two  steering  committee  meetings  were  held 
during  this  phase.     One  of  the  steering  committee  meetings 
preceded  the  pilot  test  and  one  was  held  shortly  after  it 
was  completed. 

Third  Project  Steering  Committee  Meeting 

The  third  project  Steering  Committee  Meeting  was  held 
in  early  June  1979,  at  the  mid-point  of  the  six  month  study. 
This  meeting  served  to  (1)  review  the  accomplishments  of  the 
study  to  date  and  the  remaining  project  activities  and 
schedule,  and  (2)  discuss  the  plans  for  the  pilot  test. 
Exhibit  3  was  presented  to  discuss  the  project  timing  and 
activi  ties . 

Refined  objectives  for  the  pilot  test  were  presented, 
these  included: 

o    applying  and  testing  the  performance  evaluation 
methodology ; 


!_/     Transit  systems  receiving  state  assistance  in  Michigan 
were  required  to  submit  quarterly  reports  to  UPTRAN 
including  operating  and  financial  data  between  1972  and 
1978.     These  reports  were  replaced  by  the  UMTA  Section 
15  reports  through  an  act  of  the  State  Legislature  in 
November  1978. 
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EXHIBIT  3 


TIMING  OF  ACTIVITIZS  FOR  PILOT  APPLICATION 
AND  iffiMAINDER  OF  THIS  PROJECT 


PHASE  I:        RESEARCH  AND  REPORT  ON  TRANSIT  SYSTEM  EVALUATION  METHODOLOGIES 
Complece 

PHASE  II:      DEVELOP  EVALUATION  METH0D0L0G7  (AND  DATA  NEEDS  MANUAL) 
Complece 

PHASE  III:    CONDUCT  PILOT  PERFORMANCE  EVALUATION 


Task  3:  Apply  Evaluation  Miechodo- 
log7  In  Pilot  Performance 
Evaluation 

5.1:    Select  AreaCa)  for  Pilot 

5.2:  Conduct  Peer  Comparison 
Using  Available  Section 
15  Data 

5.3:    Conduct  Time  Series 

Assessment  Using  Data 
For  Kalamazoo 

^ask  6:    Document  Pilot  Results 

*  6.1:    Document  Results 

*  6.2:    Review  Results  with 

System  Used  in  Time 
Series  Analysis 

*  6.3:    Review  Rasiilts  With 

Steering  Committee 
And  Initiate  Phase  IV 

★PHASE  IV:     PREPARE  TRANSIT  EVALUATION 
 MANUAL  

*Taslc  7:    Prepare  Transit  Evalua- 
tion Manxial  and  Train 
Staff 

*  7.1:    Prepare  Draft  Manual 

And  Training  Materials 

*  7.2:    Revise  Draft  Manual 

And  Training  Materials 

*  7,3:     Conduct  Training  Session 


*    Task  and  Subcask  Unchanged  from  Proposal 
Tentative  Meeting  Schediile 


MONTHS  TO  PROJECT  COMPLETION 
12  3 
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o  assessing  the  methodolog 
implementation,  the  time 
assess  information  and  t 


y  in  terms  of   its  ease  of 

required  to  produce  and 
he  clarity  of  results; 


o     illustrating  the  use  of  data  in  the  development  of 
the  performance  indicators; 


o    examining  the  use  and  implication  of  various  ap- 
proaches for  identifying  indicators  in  the  diag- 
nostic phase  for  more  detailed  analysis  through  peer 
comparison  and  time  series  analysis;  and 


o    exploring  the  range  of  explanations  for  differences 
among  transit  systems. 


The  planned  approach  for  conducting  the  pilot  test  was 
presented  at  this  meeting  including  a  review  of  data  avail- 
able within  Michigan  and  from  transit  systems  outside 
Michigan  that  could  be  used  in  the  test.     Since  no  objec- 
tions were  raised,  or  modifications  proposed,   the  pilot  test 
was  subsequently  conducted  as  planned. 


Pilot  Test  Procedure  and  Findings 

The  pilot  test  included  both  a  peer  comparison  and  time 
series  analysis.     The  first  phase  of  the  peer  comparison 
included  identifying  performance  indicators  one  and  two 
standard  deviations  above  and  below  the  mean  value  for  each 
indicator.     These  indicators  were  investigated  in  greater 
detail  in  the  second  phase  of  the  peer  comparison.  The 
transit  systems  were  contacted  in  the  second  phase  by 
telephone  and  the  factors  affecting  the  values  of  selected 
performance  indicators  were  discussed. 


In  all  instances,  explanations  were  available  from  the 
transit  operators  to  explain  the  values  of  the  performance 
indicators.     In  some  instances,   the  operator  explained  that 
the  data  quality  was  suspect;   these  data  were  eliminated 
from  further  analysis  and  methods  to  improve  data  quality 
were  discussed.     In  most  instances,  however,  explanations 
such  as  the  following  were  provided: 


o  the  values  of  maintenance  and  materials  and  supplies 
related  indicators  were  high  because  a  major  vehicle 
overhaul  program  had  been  conducted  in  1978; 


o     the  values  of   indicators  reflecting  wage  rates  were 
higher  in  the  northern  and  northeastern  transit 
systems  because  of  regional  wage  rates; 
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the  value  of  indicators 
were  extremely  low  in  a 
all  non-union  part-time 
benefit  package; 


measuring  fringe  benefits 
transit  system  which  used 
labor  to  minimize  its 


o     the  value  of  indicators  for  fare  revenue  were  low 
and  state  and  local  support  high  for  a  transit 
system  that  operated  as  a  publicly  supported, 
largely  free  fare  service;  and 

o     the  value  of  an  indicator  which  measures  driver 
utilization  was  low  reportedly  because  the  recent 
labor  contract  established  a  liberal  sick  leave 
policy  which  was  believed  to  be  directly  related  to 
the  94  percent  increase  in  sick  leave  between  1977 
and  1978. 


In  general,  using  the  performance  indicators  as  the 
basis  for  discussing  performance  was  found  to  be  both  easy 
and  informative,  even  though  discussions  were  conducted 
exclusively  by  telephone. 


The  time  series  analysis  was  conducted  by  first  identi- 
fying all  performance  indicators  that  had  changed  signifi- 
cantly over  time  for  the  Michigan  transit  system.  These 
indicators  were  reviewed  and  questions  developed  about 
factors  that  might  have  affected  indicator  values.     A  site 
visit  was  then  arranged  with  the  transit  system.  Since 
an  objective  of  the  test  was  to  review  the  usefulness  of  the 
indicators  included  in  the  methodology,  the  values  of  all  of 
the  performance  indicators  were  reviewed  with  this  transit 
operator  rather  than  only  discussing  those  indicators  iden- 
tified for  more  detailed  analysis  in  the  diagnostic  phase. 


Interviews  were  conducted  with  the  acting  and  previous 
general  manager,  the  maintenance  supervisor,  several  admin- 
istrative personnel  and  the  transit  system  auditors  during 
approximately  three  days.     All  of  the  questions  raised  in 
the  interviews  could  not  be  answered  because  1)  there  had 
been  a  complete  turnover  of  supervisory  and  management 
personnel  of  the  transit  system  during  the  three  years 
covered  by  the  analysis  and  2)  the  quality  of  some  of  the 
data  from  1976  and  1977  was  considered  suspect  by  the 
transit  system.     It  was  generally  concluded  that  in  the 
future,  when  data  reporting  is  more  routine,  the  analysis 
would  be  less  time  consuming,  and  easier  to  complete. 

The  results  of  the  pilot  test  including  its  objectives, 
procedures  and  findings  were  documented  after  the  pilot  test 
was  completed. 
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Fourth  Project  Steering  Committee  Meeting 


This  meeting  was  held  in  midsummer  of  1979.  The 
objectives  of  this  meeting  were  to: 

o     review  and  discuss  the  pilot  application  of  the 

performance  evaluation  methodology  for  the  mid-size 
transit  system  in  Michigan;  and 

o     intiate  Phase  IV:     Documentation  of  the  Performance 
Evaluation  Manual. 


The  presentation  regarding  the  pilot  test  of  the  methodology 
included  a  discussion  about  the: 

o    objectives  of  the  pilot  evaluation; 

o    pilot  evaluation  overview; 

o    pilot  peer  comparison:     procedures  and  findings; 

o    pilot  time  series  assessment:     procedures  and 
findings ; 

o    differences  between  the  pilot  and  future  applica- 
tions of  the  methodology; 

o    conclusions  from  the  pilot  evaluation  including 

suggested  refinements  to  the  methodology  and  apparent 
strengths  and  limitations. 


The  discussion  about  evaluation  findings  focused  on 
the  relative  ease  of  gathering  information  and  on  the 
various  factors  reported  to  have  influenced  transit  system 
performance.     Suggested  differences  between  the  pilot  and 
future  applications  of  the  performance  evaluation  methodol- 
ogy were  also  discussed  including: 

o     UMTA  Section  15  reports  should  be  complete  and  more 
accurate ; 

o     the  evaluation  process  will  become  more  routine; 

o  evaluations  should  focus  more  on  monitoring  service, 
sharing  information  and  identifying  service  improve- 
ments ; 

o    additional  documentation  and  data  will  be  available; 
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o    the  identification  of  performance  indicators  for 
detailed  review  may  be  modified  to  reduce  or  expand 
the  scope  of  the  evaluations;  and 

o     gradual  refinements  may  be  introduced  to  meet  future 
needs  of  transit  systems  and  the  state. 


Neither  of  the  meetings  held  in  this  phase  of  the 
project  were  well  attended.     Those  who  did  attend  appeared 
to  find  the  material  presented  informative  and  were  gener- 
ally supportive  of  the  study's  progress. 


PHASE  IV  -  PREPARE  TRANSIT  EVALDATION  MANUAL 

During  August  1979,  a  draft  final  report  of  the  "Evalu- 
ation Manual  for  Mid-Size  Transit  Systems  in  the  State  of 
Michigan"  was  prepared.     The  draft  manual  included  four 
sections  outlined  in  Exhibit  4. 

The  draft  final  report  of  the  Evaluation  Manual  for 
Mid-Size  Transit  Systems  in  the  State  of  Michigan  and  the 
draft  Data  Needs  Manual,  prepared  in  Phase  II  of  the  study, 
were  distributed  in  early  September  1979  to  the  project 
steering  committee  members,  the  ten  mid-size  transit  systems 
in  Michigan,  and  the  UMTA  regional  representative  respon- 
sible for  transit  activities  in  the  State  of  Michigan. 
After  a  3-week  review  period,  a  meeting  was  held  with  all 
recipients  of  the  draft  evaluation  methodology  to  discuss 
the  proposed  procedures  and  their  anticipated  application  by 
the  State. 

Final  Project  Steering  Committee  Meeting 

The  final  project  steering  committee  meeting  for  the 
project  to  develop  and  test  an  evaluation  methodology  for 
mid-size  transit  systems  in  Michigan  was  held  October  30, 
1979.     (This  meeting  included  the  project  steering  committee 
members  and  the  other  recipients  of  the  draft  Evaluation  and 
Data  Needs  manuals.) 

The  objectives  of  this  meeting  were  to: 

o    review  the  objectives  and  activities  of  the  project 
to  Develop  and  Test  an  Evaluation  Methodology  for 
Mid-size  Transit  Systems  in  Michigan; 

o    present  the  evaluation  methodology:     the  objectives, 
procedure  and  format  of  results; 
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EXHIBIT  4 


OUTLINE  OF  THE  PRATT  EVALUATION  MANUAL 

I  Introduction 
Legislative  Mandate 

Objectives  of  the  Evaluation  Manual 

II  Description  of  the  Perfonpance  Evaluation  Methodology 

The  Evaluation  Methodology:     Overview  and  Intent  of  Activities 

Peer  and  Time  Series  Comparison  of  Transit  Performance:     Current  Concerns 

Performance  Indicators  Developed  for  This  Methodology 

III  Guidelines  for  Conducting  a  Diagnostic  Evaluation  of  Transit  Performance 
Collect  Data 

List  the  Absolute  Value  of  All  Data 

Develop  the  Performance  Indicators 

Screen  and  Validate  the  Performance  Indicators 

Calculate  the  Range,  Mean,  and  Standard  Deviation  of  the  Current 
Year  and  the  Percent  Change  Since  the  Previous  Year  for  each 
Indicator 

Identify  Performance  Indicators  that  Appear  to  be  Outliers 

IV  Guidelines  for  Conducting  a  Detailed  Evaluation  of  Transit  Performance 
Prepare  for  Site  Visit (s)  with  Each  Transit  Operator 

Conduct  Site  Visit (s)  with  Each  Transit  System 
Dociiment  the  Detailed  Evaluation  Findings 
The  draft  manual  also  included  three  appendicies: 

Appendix  A  -  Phase  I  Findings:     Research  and  Report  on  Transit  System 

Evaluation  Methodologies 
Appendix  B  -  Pilot  Application  Results  of  the  Performance  Evaluation 

Methodology  for  Mid-Size  Transit  Systems  in  Michigan 
Appendix  C  -  Suggested  Inquiries  and  Issues  for  Discussion  with  Transit 
Operators  for  the  Performance  Indicators  in  the  Evaluation 
Methodology. 
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o    present  the  evaluation  methodology:     the  objectives, 
procedure  and  format  of  results; 

o    discuss  the  results  of  the  pilot  application  of  the 
methodology; 

o    discuss  the  benefits  of  the  evaluation  program;  and 

o     conclude  with  an  open  discussion  and  question  and 
answer  session. 


During  the  meeting,  representatives  of  UPTRAN  presented 
material  summarizing  the  basis  of  the  department's  authority 
to  conduct  evaluations.     This  included  citations  of  recent 
legislation,  references  to  recent  statements  made  by  Governor 
Milliken,  and  excerpts  from  the  May  1979  report  from  the 
3tate  Auditor  General. 

Similar  to  the  first  project  steering  committee  meeting 
the  discussion  in  this  meeting  was  concerned  largely  with 
policy  issues.     Several  of  the  transit  operators  were  active 
in  the  discussions.     Specifically,  these  operators  still 
questioned  the  State's  authority  to  evaluate  their  perfor- 
mance, and  expressed  dissatisfaction  with  the  use  of  peer 
comparison  as  part  of  the  initial  diagnostic  phase  of  the 
evaluation  procedures. 

The  operators  suggested  that  the  methodology  exclude 
peer  comparison  and  include  only  the  time  series  self 
assessment  element  of  the  diagnostic  phase.     In  support  of 
this  recommendation,  the  operators  discussed  the  benefits  of 
self  analysis  over  time  in  light  of  the  uniqueness  of  each 
system  and  the  drawbacks  of  peer  comparisons.     In  response 
to  this  suggestion  UPTRAN  reviewed  the  potential  benefits  of 
peer  comparison  as  a  diagnostic  tool  and  suggested  further 
analysis  of  the  diagnostic  phase  of  the  methodology  focusing 
on  the  benefits  and  limitations  of  peer  comparisons. 

Additional  concerns  raised  at  the  meeting  reflected 
the  transit  operators'   interest  in  more  extensive  involve- 
ment in  the  evaluation  efforts.     At  the  conclusion  of  the 
meeting,  the  operators  were  asked  to  participate  on  the 
project  steering  committee  of  the  proposed  upcoming  project 
to  refine  and  implement  the  evaluation  procedures.  Two 
transit  operators  volunteered  to  serve  on  the  committee. 

Following  the  meeting,  a  letter  was  sent  to  everyone 
invited  to  the  meeting.     The  letter  summarized  the  meeting's 
results;  stated  UPTRAN 's  objectives  in  conducting  evalua- 
tions,  in  general  and  those  specific  to  transit;  enumerated 
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the  intended  use  of  evaluation  results;   and  cited  the  basis 
of  UPTRAII '  s  mandate  to  conduct  performance  evaluations. 
(See  Appendix  A.) 

This  meeting  marked  the  end  of  the  project  to  Develop 

and  Test  an  Evaluation  Methodology  for  Mid-size  Transit 
Systems  in  Michigan.     Because  no  substantive  comments  were 

received  on  the  draft  Evaluation  and  Data  Needs  Manuals,  the 

manuals  were  not  revised  and  finalized.     It  was  assumed  that 

revisions  would  be  made  during  the  next  project  in  which  the 
methodology  was  to  be  refined  and  implemented. 
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V.     PROJECT  TO:      TEST  AND  REFINE  THE  EVALUATION 

METHODOLOGY  FOR  MID-SIZE  TRANSIT  SYSTEMS   IN  MICHIGAN 


Following  the  final  project  steering  committee  meeting 
for  the  project  to  Develop  and  Test  an  Evaluation  Methodology 
for  the  Mid-Size  Transit  Systems  in  Michigan  (October  30, 
1979),  a  work  plan  was  developed  for  UPTRAN  to  refine  and 
implement  the  methodology.     The  work  plan  was  circulated  to 
the  steering  committee  and  subsequently  approved  by  UPTRAN. 
In  February  1980  the  first  steering  committee  meeting  for 
this  next  project  was  held.     This  committee  included  two  new 
representatives  of  transit  systems  in  Michigan  in  addition 
to  the  members  from  the  previous  project's  committee .i/ 


FIRST  PROJECT  STEERING  COMMITTEE  MEETING 

During  this  meeting,  the  transit  operators  on  the 
project  steering  committee  again  challenged  UPTRAN 's  author- 
ity to  conduct  and  report  on  perforrsance  evaluations  of  the 
transit  sysems.     The  operators  were  particularly  concerned 
about  including  the  results  of  the  performance  evaluations 
in  the  annual  report  to  the  State  Legislature.     In  addition, 
objections  were  raised  regarding  UPTRAN 's  plan  to  formally 
institute  the  evaluation  methodology  in  the  upcoming  year. 
The  transit  operators  stated  that  final  implementation 
should  not  occur  until  the  methodology  was  pilot  tested 
statewide  (i.e.,  in  all  ten  of  the  mid-size  transit  systems 
in  Michigan)  and  further  refined.     Refinement  of  the  evalua- 
tion procedures  would  be  based  on  the  results  of  the  state- 
wide pilot  test  including  the  input  of  the  transit  operators. 
Finally,  it  was  suggested  that  the  evaluation  procedures 
should  be  more  directly  beneficial  to  the  transit  systems  as 
a  management  tool  for  self -evaluation . 

In  summary,  the  key  concerns  of  the  transit  operator 
representatives  raised  at  the  meeting  were  that: 

o     the  authority  of  UPTRAN  to  evaluate  public  transpor- 
tation performance  should  be  further  clarified 
possibly  through  direct  inquiry  to  the  state  legis- 
lature; 


\_l     One  of  the  members  from  the  previous  project's  steering 
committee  had  changed  employment  from  Tri-County  Plan- 
ning Commission  to  Capital  Area  Transportation  Author- 
ity, Lansing,  Michigan.     Therefore,   the  new  project 
steering  committee  included  three  transit  operator 
representatives  in  addition  to  the  representative  from 
the  MPTA. 
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o     the  format  and  content  of  any  report  developed  in 
whole  or  part  from  the  results  of  the  evaluations  be 
developed  in  conjunction  with  the  transit  operators 
and  subject  to  their  review  before  it  would  be  dis- 
tributed (in  particular  to  the  Michigan  State 
Legislature) ; 

o    a  state-wide  pilot  test  of  the  evaluation  method- 
ology should  be  conducted.     An  objective  of  this 
test  would  be  the  critical  review  of  the  usefulness 
of  information  gathered  through  a  peer  comparison  of 
the  transit  systems  in  Michigan;  and 

o     the  products  of  this  project  more  directly  benefit 
the  transit  operators  by  providing  a  tool  explicitly 
for  self  evaluation. 


PROJECT  WORK  PLAN  REVISIONS 

In  response  to  the  concerns  of  the  transit  operators, 
UPTRAN  altered  its  plans  as  to  the  way  the  evaluation 
procedures  would  be  implemented.     It  was  agreed  that: 

o     the  evaluation  methodology  would  be  tested  on  a 
statewide  basis  (i.e.,  in  each  of  the  ten  mid-size 
transit  systems),  and  refined  with  the  input  of  an 
expanded  project  steering  committee  and  the  results 
of  the  statewide  pilot  test; 

o    a  task  would  be  included  in  the  revised  work  plan  to 
develop  the  draft  format  and  suggested  content  of 
the  report  to  the  Michigan  State  Legislatuare  on  the 
performance  and  progress  of  the  mid-size  transit 
systems  in  Michigan.     A  preliminary  draft  report, 
for  discussion  purposes,   would  be  developed  early  in 
the  project  so  that  each  element  of  the  draft  report 
could  be  reviewed,  modified,  developed  in  more 
detail,  or  eliminated  during  the  remainder  of  the 
project;  and 

o    a  manual  for  transit  self -evaluation  would  be 

developed  for  the  use  and  benefit  of  the  mid-size 
transit  systems  in  Michigan.     The  methodology  and 
techniques  developed  for  UPTRAN  (in  the  previous 
project)  would  be  adapted  and  expanded  to  serve 
transit  management  by  improving  service  monitoring 
and  evaluation  procedures. 
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The  revised  work  plan  for  the  project  to  Test  and  Refine  the 
Evaluation  Methodology  for  Mid-size  Transit  in  Michigan, 
summarized  in  Exhibit  5,  was  prepared  and  approved  by  UPTRAN 
and  the  State  of  Michigan  in  August  1980.     It  is  anticipated 
that  this  project  will  extend  for  12  to  18  months.  The 
statewide  pilot  test  and  the  preparation  of  the  self-evalu- 
ation manual  should  be  completed  by  December  1981. 


PHASE   I:      TRANSITION  TO  REFINEMENT  AND  STATEWIDE  TESTING  OF 
AN  EVALUATION  METHODOLOGY  FOR  MID-SIZE  TRANSIT  IN  MICHIGAN 

The  first  phase  in  the  revised  project  to  Test  and 
Refine  the  Evaluation  Methodology  for  the  Mid-Size  Transit 
Systems  in  Michigan  was  designed  to  serve  as  a  transition 
from  the  initial  project  which  had  been  completed  in  the 
fall  of  1979  (in  which  the  methdology  was  developed)   to  the 
current  one.     One  of  the  first  activities  in  the  transition 
phase  was  to  interview  the  steering  committee  members, 
focusing  on  transit  system  representatives.     The  primary 
area  of  discussion  raised  during  these  interviews  related  to 
the  need  for  UPTRAN  to  prepare  a  formal  policy  statement  in 
which  it  would  state  the  objectives  of  the  transit  evalua- 
tions and  the  intended  use  of  evaluation  results.  These 
concerns  reflected  the  transit  operator's  interest  in 
further  clarifying  UPTRAN' s  authority  to  evaluate  their 
performance  and  defining  the  extent  to  which  the  evaluations 
may  affect  the  transit  systems'  management  and  operating 
activities  and  finances. 

In  September  1980,  a  statement  was  prepared  in  response 
to  the  interview  findings  and  to  resolve  past  challenges  to 
UPTRAN 's  authority  posed  at  project  meetings.     This  state- 
ment was  distributed  for  review  first  within  UPTRAN  and  then 
to  the  project  steering  committee  members.     (See  Appendix 
B.)     On  November  3,   1980,   the  policy  statement  was  discussed 
at  a  project  steering  committee  meeting.     The  statement  was 
generally  accepted  and  challenges  to  UPTRAN 's  authority  to 
evaluate  public  transportation  performance  have  since 
subsided . 

At  this  meeting,  it  was  also  decided  that:     1)  the 
development  of  the  draft  format  and  suggested  content  for 
the  annual  report  to  the  Michigan  legislature  would  be 
removed  from  the  scope  of  the  project,  and  2)  that  the 
project  would  move  forward  as  planned  with  the  statewide 
pilot  testing  efforts  and  development  of  the  self -evaluation 
manual . 

While  it  was  agreed  that  UPTRAN  would  continue  to 
modify  and  improve  its  report  to  the  legislature,  members  of 
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EXHIBIT  5 

TEST  AND  REFINE  THE  EVALUATION 
METHODOLOGY  FOR  MID  SIZE  TRANSIT  IN  MICHIGAN 
PRELIMINARY  PROJECT  SCHEDULE 


PHASfS  ANO  TASKS 


1         2        3        4        5        8  7 


MONTHS  FnpM  PWQJECT  INITIATON 


9        10       11  12 


PfWM  I     Tnmitlon  Par  Ra^lnwnant  And  Stit«wrid«  TMtIng  Of 
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the  steering  committee,  in  particular  those  representing  the 
transit  systems,  believed  that  these  activities  should  not 
be  part  of  the  project  to  Test  and  Refine  the  Evaluation 
Methodology.     No  detailed  explanation  was  provided  or 
pursued  regarding  why  the  development  of  the  draft  format 
and  suggested  content  of  the  annual  report  to  the  state 
Legislature  was  being  removed  from  the  project  scope. 

Since  this  steering  committee  meeting,  the  project  has 
moved  forward,  focusing  largely  on  the  statewide  pilot  test 
activities. 
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VI.     LESSONS  LEARNED   IN  THE  DEVELOPMENT 
OF  THE  EVALUATION  METHODOLOGY 


Two  major  lessons  were  learned  during  the  development 
of  the  evaluation  methodology  for  mid-size  transit  systems 
in  Michigan.     First,   there  is  a  need  to  establish  the 
groundwork  for  conducting  evaluations  of  public  transporta- 
tion system  performance.     Second,   the  concerns  and  perspec- 
tives of  the  public  transportation  systems  regarding  perfor 
mance  evaluation  must  be  addressed  and  integrated  into  the 
development  and  implementation  of   the  evaluation  program. 


IMPORTANCE  OF  ESTABLISHING  THE  GROUNDWORK  FOR  CONDUCTING 
EVALUATIONS  OF  PUBLIC  TRANSPORTATION  PERFORMANCE 

It  is  important  that  a  state  or  agency  interested  in 
evaluating  public  transportation  performance  establish  the 
groundwork  for  conducting  evaluations.     Such  efforts  will 
improve  the  state  or  funding  agency's  effectiveness  by:  1) 
enhancing  its  ability  to  develop  an  evaluation  program,  and 
2)  facilitating  the  timely  implementation  of  evaluation 
procedures.     The  issues  that  should  be  addressed  and  activi 
ties  that  should  be  performed  to  establish  the  groundwork 
for  conducting  the  evaluations  include: 

o    establishing  the  basis  of  authority  to  conduct 
performance  evaluations; 

o     defining  the  objectives  for  the  evaluation  and 

specifying  the  intended  use  of  evaluation  results; 

o    developing  an  evaluation  methodology  and  evaluation 
process ; 

o    preparing  for  the  implementation  of  the  evaluation 
program;  and 

o    communicating  the  above  information  to  the  public 
transportation  systems. 


Exhibit  6  summarizes  important  considerations  including  the 
timing  of  each  of  these  five  activities.     Each  activity  is 
described  below  in  more  detail,  highlighting  the  experience 
within  the  State  of  Michigan. 
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TIMING 

These  activities  follow  the  development  of  the  evaluation 
program  and  precede  its  full  scale  implementation.  The 
amount  of  time  required  will  be  influenced  by: 
.  the  number  of  transit  systems  effected  by  the 

evaluation  program; 
.  the  familiarity  of  the  evaluators  and  the  transit 
systems  with  the  evaluation  objectives,  procedures 
and  uses  of  results;  and 
.  the  complexity  of  the  evaluation  methodology 
including  data  requirements. 
A  reasonable  benchmark  is  one  year. 

The  timing  and  frequency  of  communication  with 
transit  systems  will  vary  among  areas  developing  and 
implementing  transit  evaluation  programs.  In  some 
areas,  some  or  all  of  the  transit  operators  may  be 
Involved  throughout  the  development  and  implemen- 
tation activities.  At  a  minimum,  all  operators  that 
will  be  evaluated  should  be  fully  informed  about  the 
evaluation  program  before  it  is  implemented. 

CONSIDERATIONS 

These  activities  serve  as  a  traruition  from  the  development  to  the  implementation  of 
the  evaluation  program.  Adequate  testing  and  refinement  of  the  evaluation  method- 
ology (i.e.,  analytical  techniques)  and  the  evaluation  process  (i.e.,  roles  and  respon- 
sibilities of  participants,  timing,  etc.).  In  addition,  there  should  be  training  by  hands 
on  experience  for  the  evaluators  and  detailed  debriefing  of  the  transit  operators  not 
yet  familiar  with  the  evaluation  program. 

Communication  with  the  transit  systems  that  will  be  evaluated  is  important  to  the 
successful  implementation  of  the  evaluation  program.  The  frequency  and  nature  of 
communication  with  the  transit  systems  will  be  determined,  in  part,  by  the  role  the 
operators  play  in  the  development  of  the  eveluation  program.  The  more  integral  the 
operators  are  in  determining  the  objectives  of  the  evaluatioru,  uses  or  results,  and 
procedures  used,  the  more  involved  they  will  be.  Regardless  of  the  level  of  operator 
involvement  their  concerns  and  recommendations  should  be  coiuidered  and  any 
requested  information  should  be  made  readily  available. 

ACTIVITIES 

Prepare  for  the  Implementation  of  the 
Evaluation  Program 

Communicate  with  Transit  Systems 
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Establishing  the  Basis  of  Authority 


A  state  or  funding  agency  desiring  to  initiate  an 
evaluation  program  must  first  establish  and  present  the 
basis  of  its  authority  to  conduct  performance  evaluations. 
This  requires  a  careful  review  of  the  statements  that 
specifically  mandate  or  infer  that  performance  evaluations 
should  be  conducted  by  the  agency.     As  a  first  step,  agree- 
ment should  be  reached  within  the  organization  initiating 
the  evaluation  program  regarding  its  responsibilities  and 
authority . 

When  this  basis  of  authority  is  clearly  defined  through 
legislative  or  executive  order,  or  is  specifically  deline- 
ated in  the  organization's  charter  of  responsibilities,  it 
will  help  reduce  misunderstandings.     Alternatively,  if  the 
basis  of  authority  is  loosely  structured,  it  will  generally 
be  subject  to  interi    etation  and  can  be  challenged. 

If  necessary,  clarification  of  intent  should  be  sought 
from  the  source  that  initiated  the  evaluation  requirements 
(i.e.,  the  legislature,  city  council,  executive  branch, 
etc.).     In  these  instances,   it  may  be  appropriate  to  prepare 
a  policy  statement  on  administrative  procedures  that  clarify 
the  responsibilities  and  authority  of  the  state  or  agency 
with  regard  to  performance  evaluation. 

In  the  State  of  Michigan,  it  was  necessary  for  UPTRAN 
to  prepare  a  policy  statement  to  clarify  its  authority  to 
evaluate  public  transportation  in  the  state. As  discussed 
in  Section  1 1  of  this  report,   the  legislative  and  executive 
orders,   considered  by  UPTRAN  as  a  mandate  to  conduct  evalua- 
tions and  annually  report  on  the  performance  of  public 
transportation  programs,   require  the  measurement  of  public 
transportation  efficiency  and  effectiveness,  but  do  not 
specifically  stipulate  that  the  state  develop  and  implement 
an  evaluation  program.     Rather,  the  requirements  were 
inferred  from  the  legislation  and  statements  made  by  the 
governor,  and  the  State  Auditor  General's  May  1979  report. 

During  the  development  of  the  evaluation  program, 
UPTRAN 's  authority  was  questioned  by  the  transit  systems 
subject  to  evaluation.     The  statement  prepared  by  UPTRAN  in 
November,   1980  was  generally  accepted,  and  since  its  distri- 
bution challenges  to  UPTRAN 's  authority  to  evaluate  transit 
in  the  state  have  subsided. 


1_/     This  document  also  served  to  clarify  the  objectives  of 
the  evaluations  and  intended  uses  of  evaluation  results. 
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Defining  the  Objectives  and   Intended  Uses  of  Evaluation 
Results 

A  state  or  funding  agency  initiating  a  performance 
evaluation  program  should  define  the  objectives  of  the 
evaluations  and  intended  uses  of  evaluation  results.  This 
information  is  of  prime  importance  both  to  the  public 
transportation  systems  subject  to  evaluation  and  to  the 
state  or  agency  developing  the  evaluation  program.  The 
information  communicates  to  the  transit  systems  why  the 
evaluations  are  being  conducted  and  how  the  systems  may  be 
affected.     The  uncertainty  of  not  knowing  the  potential 
effects  of  evaluations  could  be  unsettling  and  potentially 
threatening  to  the  systems  subject  to  evaluation. 

The  objectives  of  the  evaluation  and  intended  uses  of 
evaluation  results  are  important  to  the  state  or  agency 
responsible  for  the  evaluation  program  because  they  estab- 
lish the  framework  for  the  evaluation  procedures.     It  is 
necessary  to  know  first  what  you  must  accomplish  before  a 
methpd  can  be  defined,  i.e.,  the  methodology  should  be 
developed  to  fit  predefined  objectives  and  uses  of  evalua- 
tion results. 

One  of  the  first  steps  in  the  project  conducted  by 
UPTRAN  to  Develop  and  Test  an  Evaluation  Methodology  for 
Mid-Size  Transit  in  Michigan  was  the  development  of  evalua- 
tion objectives.     The  objectives  included  the  systematic  and 
routine  review  of  performance  to: 

o     increase  the  understanding  about  public  transporta- 
tion operations  and  performance  in  the  State  of 
Michigan  by  the  State  Department  of  Transportation, 
the  State  Legislature,  and  other  interests; 

o     facilitate  the  exchange  of  information  among  public 
transportation  systems  particularly  in  areas  where 
there  are  innovative  operations  and  outstanding 
performance; 

o    monitor  the  use  of  public  funds  for  public  transpor- 
tation, service  development  and  delivery;  and 

o     identify  opportunities  to  improve  performance  by 
promoting  more  efficient  and  effective  transit 
services . 


Specification  of  the  intended  uses  of  evaluation 
results,   generally  complementing  the  objectives,   was  also 
made,  but  was  somewhat  less  definitive.     UPTRAN  stated  that 
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the  evaluation  results  would  serve  as  an  important  new 
source  of  information  available  to  the  legislature,  the 
state  and  the  public  transportation  systems.     UPTRAN  also 
indicated  that  it  intended  to  use  the  information  as  input 
to  its  annual  report  on  public  transportation  performance 
and  progress  required  by  Act  51.     Other  potential  uses  of 
evaluixtion  results  were  not  clearly  defined,  but  it  was 
apparent  that  over  time,  additional  uses  might  be  considered. 

Developing  an  Evaluation  Methodology  and  Evaluation  Process 

The  distinction  between  an  evaluation  methodology  and 
an  evaluation  process  is  that  the  methodology  includes  the 
analytical  techniques  while  the  process  defines  the  timing 
of  activities,  roles  of  participants,  and  related  concerns 
associated  with  implementation.  Both  the  methodology  and 
the  process  should  be  developed  to  meet  the  objectives  of 
conducting  the  evaluation  and  support  the  intended  uses  of 
the  evaluation  results. 

The  initial  project  in  Michigan  focused  on  the  devel- 
opment of  the  methodology  with  little  emphasis  on  the 
evaluation  process.     Important  findings  of  the  project  in 
which  the  evaluation  methodology  was  developed  were  that  the 
time  requirements  to  develop  an  evaluation  process  are 
substantial  and  that  the  evaluation  process  should  not  be. 
overlooked  or  considered  secondary  to  the  evaluation  method- 
ology. 

Development  of  the  Evaluation  Methodology 

During  the  development  of  the  evaluation  methodology, 
it  is  important  that  the  state  or  agency  identify  current, 
ongoing  activities  that  may,  in  part,  meet  the  objectives  of 
the  evaluation.     Similarly,  every  effort  should  be  made  to 
make  use  of  available  information  to  minimize  redundancy. 
It  is  likely  that  some  information  useful  for  evaluation 
purposes  will  be  readily  available.     This  may  include 
information  prepared  by  the  public  transportation  system, 
the  state,  or  other  agencies  to  meet  other  existing  require- 
ments.    These  requirements  may  include  local,  state,  or 
federal  planning,  service  monitoring,  audit,   grant  applica- 
tion, or  reporting  requirements. 

Use  of  existing  information  has  several  benefits.  It 
may  reduce  the  staff  requirements  for  carrying  out  the 
evaluations  both  for  the  public  transportation  system  and 
the  state  or  agency.     Using  existing  information  may  also 
reduce  the  elapsed  time  before  findings  and  conclusions  can 
be  developed  from  the  evaluations. 
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To  the  extent  possible,   the  analytical  techniques 
included  in  the  evaluation  methodology  should  be  straight- 
forward and  easily  understood.     Simple  analytical  techniques 
will  lend  to  the  acceptance  of  the  methodology  and  ease  of 
training  people  to  conduct  the  evaluations  and  implement  the 
evaluation  procedures. 

These  guidelines  served  as  the  basis  for  the  development 
of  the  evaluation  methodology  in  Michigan.  Importantly, 
data  required  under  UMTA  Section  15  (financial  and  operating 
statistic)  reporting  requirements  is  the  basis  for  the 
diagnostic  phase  of  the  evaluation.     This  serves  to  minimize 
data  gathering  by  the  transit  systems  and  should  insure  that 
data  is  routinely  available  to  UPTRAN. 

Development  of  the  Evaluation  Process 

The  evaluation  process  that  is  established  should 
reflect  the  capabilities  and  interests  of  the  participants 
in  the  evaluation  program.     For  example,   the  planned  timing 
of  activities  should  liberally  allow  for  the  current  sched- 
ule and  staff  commitments  of  the  public  transportation 
systems  and  state  or  agency  personnel  involved.     An  overly 
ambitious  schedule  is  likely  to  be  unactfievable ,  particu- 
larly during  the  initial  years  in  which  the  evaluations  are 
performed.     Adequate  review  periods  of  preliminary  evalua- 
tion findings  should  be  incorporated  in  the  evaluation 
program  to  allow  for  rebuttal  by  the  evaluated  systems,  as 
appropriate.     This  feedback  enhances  the  likelihood  that  the 
evaluation  results  will  accurately  reflect  the  situation  of 
the  public  transportation  systems. 

An  evaluation  process  sensitive  to  these  concerns  is 
currently  being  developed  in  Michigan  for  the  mid-size 
transit  systems.     The  statewide  pilot  test  activities  will 
include: 

o    assessment  of  the  time  requirements  of  each  step  of 
the  evaluations  together  with  an  assessment  of  the 
schedule  and  time  commitments  of  the  transit  opera- 
tors; 

o    development,   testing  and  discussion  of  the  roles  and 
responsibilities  of  UPTRAN  and  transit  system 
staff; 

o     interaction  among  UPTRAN,   the  transit  operators  and 
the  steering  committee,  throughout  the  pilot  test 
focusing  on  critical  milestones  including  the 
initiation  and  completion  of  the  diagnostic  and 
detailed  phases  of  the  evaluations;  and 
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o    review  and  discussion  of  evaluation  findings  before 
the  results  are  documented  or  presented. 


Preparing  for  the  Implementation  of  the  Evaluation  Program 

The  development  of  an  evaluation  program  should  be 
followed  by  a  period  of  transition  before  implementation. 
The  objectives  of  the  transition  period  are  to  1)  allow  for 
a  final  testing  and  refinement  of  the  methodology  and  the 
evaluation  process,  and  2)  ensure  that  all  participants  are 
trained  and  informed,  as  appropriate. 

Testing  an  evaluation  program  is  essential  because  the 
ability  to  meet  objectives  will  be  uncertain.  Refinements 
and  tailoring  of  the  evaluation  methodology  and  process  will 
probably  result  from  testing.     Adequate  training  of  state  or 
agency  staff  involved  in  the  evaluation  program  is  also 
essential.     The  transition  period  offers  an  important 
opportunity  for  this  training.     By  practicing  in  a  test 
environment,  direct  experience  in  conducting  the  evaluations 
can  be  obtained. 

The  time  required  to  test  the  evaluation  methodology, 
introduce  it  to  the  public  transportation  systems,  and  train 
state  or  agency  staff  in  its  application  will  be  influenced 
by: 

o    the  number  of  systems  to  be  evaluated; 

o    the  familiarity  of  the  systems  with  the  objectives 
of  the  evaluation,  the  use  of  results,  and  the 
evaluation  procedures;  and 

o    the  complexity  of  the  evaluation  methodology, 
including  data  requirements. 


A  period  of  one  year  is,  in  general,  a  reasonable  benchmark. 

The  statewide  pilot  testing  in  Michigan  will  serve  as 
the  transition  period  before  the  evaluation  program  for 
mid-size  transit  systems  is  implemented.     Specific  elements 
of  the  methodology  have  been  identified  by  UPTRAN,  the 
steering  committee,  and  the  transit  operators  for  further 
testing  and  refinements.     During  the  pilot  test,  UPTRAN 
staff  is  being  trained  in  all  phases  of  the  evaluation;  and 
the  transit  operators  are  being  more  fully  informed  about 
the  evaluation  activities.     A  twelve  to  eighteen  month 
period  of  transition  is  currently  planned. 
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Communicating  with  Public  Transportation  Systems 


Communication  with  the  public  transportation  systems  is 
important  for  establishing  the  groundwork  for  a  state  or 
agency  to  evaluate  performance.     Communication  between  the 
evaluator  and  the  systems  subject  to  evaluation  may  occur  at 
each  stage  of  the  development,   testing  and  ultimate  imple- 
mentation of  the  evaluation  program.     To  a  great  extent,  the 
public  transportation  systems'   role  in  the  development  and 
implementation  of  the  evaluation  procedures  determines  the 
nature  and  frequency  of  communication  between  the  systems 
and  the  state  or  agency. 

In  some  areas,   some  or  all  of  the  systems  that  will  be 
evaluated  may  be  directly  involved  in  the  decision  making  to 
develop  and  implement  the  evaluation  program.     In  other 
areas,  communication  with  the  systems  may  be  more  informa- 
tional.    Regardless  of  the  systems'   role,  however,   lines  of 
communication  should  be  established  so  that  operator  con- 
cerns and  recommendations  are  considered  and  requested 
information  is  readily  available.     At  a  minimum,  all  systems 
that  will  be  evaluated  should  be  fully  informed  about  the 
evaluation  program  before  it  is  implemented.     (This  effort 
should  begin  with  a  discussion  of  the  objectives  and  intended 
uses  of  evaluation  results.) 

Representatives  of  the  mid-size  transit  systems  in 
Michigan  were  involved  in  the  development  and  testing  of  the 
evaluation  methodology  from  the  outset  of  the  project. 
Initially,  a  member  of  the  MPTA  served  on  the  project 
steering  committee  as  the  representative  of  all  of  the 
raid-size  transit  systems.!./     In  retrospect,  UPTRAN  believes 
that  the  project  steering  committee  should  have  included 
representatives  from  the  transit  systems  in  addition  to  the 
MPTA.     The  MPTA  serves  largely  as  a  forum  for  the  transit 
systems  and  acts  as  their  lobbyist  in  the  state  legislature. 
Greater  technical  expertise  in  evaluation  and  closer  coordi- 
nation with  the  transit  operators  could  have  facilitated  the 
development  of  the  methodology  and  the  period  of  transition 
to  implementation. 


!_/     As  is  stated  in  Section  II  of  this  report,   UPTRAN  had 
originally  contacted  the  Capital  Area  Transportation 
Authority   (CATA)  requesting  its  participation  on  the 
Committee.     The  invitation  was  declined,  and  UPTRAN  was 
informed  that  the  MPTA  could  serve  as  the  representative 
for  the  transit  systems. 
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The  current  project  to  Test  and  Refine  the  Evaluation 
Methodology  (developed  in  the  1979  study)  includes  1)  a 
steering  committee  with  representatives  from  three  of  the 
mid-size  transit  sysxems  (each  of  whom  volunteered  to 
participate  on  the  committee),  and  2)  regularly  scheduled 
meetings  with  representatives  of  all  of  the  mid-size  transit 
systems.     An  objective  of  this  project  is  to  improve  com- 
munication with  the  mid-size  transit  systems  with  regard  to 
the  evaluation  methodology  and  its  application. 


PUBLIC  TRANSPORTATION  SYSTEM  CONCERNS  AND  PERSPECTIVES 
REGARDING  PERFORMANCE  EVALUATION 

The  concerns  and  perspectives  of  public  transportation 
systems  regarding  the  evaluation  of  their  performance  by 
states  or  other  funding  agencies  will  vary  from  system  to 
system.     In  general,  however,  the  following  questions  are 
likely  to  be  raised: 

o    How  will  the  evaluation  results  be  used? 

o    Will  the  allocation  of  funding  be  affected  by  the 
results  of  the  evaluations?     If  so,  how? 

o    What  will  the  evaluations  require  of  the  public 

transportation  systems  in  terms  of  staff  and  manage- 
ment time  and  data? 

o    Who  will  conduct  the  evaluations? 

o    What  recourse  does  a  system  have  if  it  disagrees 
with  the  evaluation  conclusions? 

o    What  benefits  can  be  expected  from  the  evaluation 
results? 

o     Is  state  or  funding  agency  involvement  in  public 
transportation  system  management  and  operation 
exceeding  established  limits  of  authority  and 
responsibility? 

o    Are  peer  comparisons  valid? 

o    Are  the  proposed  evaluation  procedures  redundant 

with  current  reporting  or  program  monitoring  activi- 
ties for  the  system? 


Simply  stated,  a  primary  concern  of  public  transporta- 
tion systems  is  how  they  may  be  affected  by  evaluations  of 
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their  performance  by  an  outside  agency.     Specifically,  the 
systems  are  concerned  about  how  the  evaluations  may  affect: 

o    financial  resources; 

o    staff  resources; 

o    control  of  management  over  internal  operations  and 
decision-making;  and 

o     local  officials'  and  the  public's  perception  of  the 
system  and  its  management. 


These  are  legitimate  concerns  because  the  performance 
evaluations  results  potentially  have  far-reaching  effects  on 
the  public  transportation  systems. 

An  important  responsibility  of  the  state  or  agency 
implementing  a  performance  evaluation  program  is  to  antici- 
pate and  consider  the  concerns  and  perspectives  of  the 
public  transportation  systems  subject  to  evaluation.  Also, 
the  mutual  responsibility  of  systems  and  the  state  or 
funding  agency  for  the  provision  of  public  transportation 
service  must  be  recognized. 

Exhibit  7  summarizes  the  concerns  of  public  transporta- 
tion systems  regarding  their  evaluation  by  an  outside  agency 
and  presents  the  suggested  role  of  the  evaluator.     Each  of 
these  concerns  was  raised  by  the  mid-size  transit  operators 
in  Michigan  and  has  been  discussed  throughout  this  report. 
Similar  issues  arose  in  Pennsylvania  and  California  when 
statewide  transit  evaluations  were  initiated. 

The  concerns  summarized  on  Exhibit  7,  are  discussed  in 
more  detail  below  together  with  the  suggested  role  of  the 
evaluators . 

Concerns  for  Financial  Resources 

Possibly  the  greatest  single  concern  of  public  trans- 
portation systems  is  how  the  results  of  a  funding  agency's 
evaluations  of  their  performance  will  affect  the  stability 
of  their  financial  assistance  from  that  agency.     There  is 
not  only  concern  that  the  amount  of  funds  available  could  be 
reduced  but  also  that  requirements  could  be  introduced  that 
restrict  or  direct  the  use  of  available  funds.     This  concern 
was  continually  expressed  by  the  mid-size  transit  operators 
in  Michigan. 
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The  potential  effects  of  an  evaluation  program  on 
financing  should  be  determined  at  the  earliest  possible  date 
and  communicated  by  the  evaluator  to  the  systems  subject  to 
evaluation.     This  enables  the  systems  to  initiate  the 
necessary  actions  to  avoid  or  minimize  a  loss  of  funds.  To 
the  extent  possible,   it  is  suggested  that  the  evaluation 
program  focus  on  service  improvement  and  accountability,  not 
penalty . 

Concerns  for  Public  Transportation  System  Staff  Resources 

A  concern  related  to  reductions  in  financial  resources 
is  the  use  of  staff  resources  in  conducting  performance 
evaluations.     Public  transportation  operators,  particularly 
those  that  operate  small  systems  and  have  limited  staff,  are 
concerned  about  the  extent  that  the  evaluations  of  perfor- 
mance will  involve  their  employees.     Many  systems  are  likely 
to  express  concern  about  any  additional  requirements  because 
they  often  are  understaffed  and  cannot  afford  additional 
staff  to  perform  new  activities. 

During  the  development  of  the  evaluation  procedures, 
the  evaluator  should  seek  to  minimize  the  time  demands  on 
public  transportation  system  personnel.     The  evaluation 
procedures  should  make  maximum  use  of  available  information, 
avoid  redundant  requirements,  and  be  simple  and  straight- 
forward.    Once  the  evaluation  procedures  are  developed, 
consideration  of  the  current  schedule  and  time  commitments 
of  the  systems  should  be  integral  to  conducting  the  evalua- 
tions . 

Concerns  for  Management  Control  over  Public  Transportation 
Operations  and  Decision  Making 

Public  transportation  systems  may  be  concerned  about 
how  the  introduction  of  routine  performance  evaluations  by 
an  outside  agency  will  affect  the  traditional  autonomy  or 
control  of  management  over  operations  and  decision-making. 
Historically,  funding  agencies  have  acted  largely  as  finan- 
cial stewards.     They  have  administered  the  distribution  of 
funds  and  ensured  accountability. 

Representatives  of  the  mid-size  transit  operators  in 
Michigan  have  repeatedly  expressed  their  concerns  about 
extending  the  role  of  the  state  into  performance  evaluation. 
While  there  is  agreement  between  UPTRAN  and  the  transit 
operators  on  the  need  for  accountability  and  financial 
stewardship,   there  is  concern  over  the  extent  that  UPTRAN 
may  become  involved  with  transit  management  and  operations 
through  the  evaluation  process. 
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Performance  evaluations  may  lead  to  recommendations  for 
improved  procedures  or  management  practices  that  could 
extend  the  role  of  outside  agencies  more  directly  into  the 
internal  management  of  the  system.     In  general,  however, 
while  states  and  other  outside  agencies  have  a  commitment  to 
improved  performance  and  the  efficient  and  effective  use  of 
public  funds,   they  do  not  want  to  get  into  the  business  of 
public  transportation  system  management.     Evaluations  are 
not  intended  to  extend  the  role  of  the  evaluator  into  the 
role  of  the  public  transportation  manager.     Rather,  manage- 
ment responsibilities  and  authority  should  be  maintained 
within  the  public  transportation  system. 

Concerns  for  the  Perception  of  the  Public  Transportation 
System  and  its  Management 

An  important  concern  of  public  transportation  systems 
is  that  performance  evaluations  might  produce  incomplete  or 
misleading  information  that  could  be  misused.     This  argument 
is  often  made  in  relation  to  the  use  of  peer  comparison  as 
an  analytical  technique. 

A  related  concern  is  that  as  a  result  of  a  performance 
evaluation,  a  system  could  look  bad  to  its  local  board,  the 
public,  or  in  comparison  to  the  other  systems  being  evalu- 
ated.    This  is  based  in  part  on  the  general  belief  that  an 
underlying  objective  of  performance  evaluation  is  to  dis- 
credit the  evaluated  systems  by  looking  for  problems  or 
performance  deficiencies  rather  than  achievement  or  progress. 
This  concern  can  be  exacerbated  if  an  adversary  relationship 
exists  between  the  evaluated  system  and  the  outside  agency 
initiating     the  evaluation  effort.     In  these  instances,  the 
system  is  likely  to  be  reluctant  to  participate  in  the 
evaluation  program  and  may  prolong  the  initiation  of  the 
evaluations . 

Even  though  it  would  probably  be  impossible  to  dispel 
all  the  concerns  of  public  transportation  systems  about  the 
intent  of  performance  evaluations,  it  is  important  for  the 
evaluator  to  promote  goodwill  and  to  create  as  objective  and 
fair  an  evaluation  program  as  possible.     Substantial  oppor- 
tunity for  input  by  the  evaluated  systems  should  be  integral 
to  the  ongoing  evaluations.     Such  efforts  are  currently 
being  pursued  in  Michigan  and  appear  to  be  successful. 

The  evaluation  process  may  incorporate  elements  explic- 
itly intended  to  protect  the  public  transportation  systems 
and  therefore  reduce  many  concerns.     For  example,  the 
process  could  require  that  the  systems  review  all  evaluation 
results  before  they  are  distributed.     In  addition,  at  the 
systems'  prerogative,  the  opportunity  for  a  formal  rebuttal, 
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to  the  findings  or  recommendations  of  the  evaluation  could 
be  included.     Most  importantly,  throughout  the  evaluations, 
equal  effort  should  be  given  to  identifying  the  progress  and 
strengths  of  each  evaluated  system  as  well  as  its  opportuni- 
ties to  improve. 
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VII.  CONCLUSION 


The  experiences  gained  in  Michigan  in  the  development 
of  a  performance  evaluation  program  for  public  transporta- 
tion are  presented  to  assist  others  to  develop  their  program 
more  effectively  and  expeditiously.     UPTRAN's  project  to 
develop  an  evaluation  methodology  for  its  mid-size  transit 
systems  lends  important  documentation  for  use  by  the  already 
growing  number  of  states,   other  funding  agencies,  and  others 
interested  in  considering  performance  evaluation  programs. 

The  Michigan  experience  confirms  that  the  process  for 
an  outside  agency  to  develop  and  implement  an  evaluation 
program  is  time-consuming  and  sensitive.     Two  important 
lessons  were  learned  through  the  experience  gained  in 
Michigan.     First,   it  is  important  that  the  groundwork  for 
evaluations  be  carefully  established.     The  development  of 
an  evaluation  methodology  represents  only  one  of  a  series  of 
important  activities  to  establish  this  groundwork.  Develop- 
ing the  methodology  must  be  preceded  by  establishing  1)  the 
authority  to  evaluate,  2)  the  objectives  of  the  evaluation, 
and  3)  the  intended  uses  of  evaluation  results.  Having 
accomplished  these  activities,   the  evaluation  methodology 
can  be  developed.     Once  developed,  the  methodology  and 
process  for  its  application  must  be  tested  and  refined  in 
conjunction  with  the  transit  systems  that  will  be  evaluated. 

The  second  important  lesson  learned  in  Michigan  is  that 
the  concerns  and  perspectives  of  the  public  transportation 
systems  subject  to  evaluation  must  be  recognized  and  con- 
sidered.    Throughout  the  development  and  implementation  of 
an  evaluation  program,   it  is  important  that  the  organization 
initiating  the  evaluations  communicates  with  the  public 
transportation  systems  and  is  responsive  to  their  concerns 
and  perspectives. 


46 


APPENDIX  A 


A.l 


STATE  OF  MICHIGAN 


WILLIAM  G.  MILLIKEN,  GOVERNOR 

DEPARTMENT  OF  TRANSPORTATION 

TRANSPORTATION   BUILDING,   425  WEST  OTTAWA        PHONE  517-373-2090 
POST  OFFICE   BOX   30050,   LANSING.   MICHIGAN  48909 

JOHN   P.  WOODFORD,  DIRECTOR 

November  1979 


TO:  Participants  at  the  Forum  on  Transit  Evaluation  Held  on  October  30, 

1979 


Thank  you  for  attending  the  forum  on  transit  evaluation  held  on  October  30,  1979. 
The  comments  mode,  questions  asked,  information  shared  and  opinions  expressed  in 
the  forum  helped  clear  the  air  and  suggested  constructive  actions. 

The  primary  intent  of  this  letter  is  to  define  what  the  Bureau  will  do  and  what  the 
Bureau  needs  by  way  of  cooperation  from  transit  operators  and  others.  This  letter 
will  also  state  the  Bureau's  objectives  for  doing  transit  evaluation  and  the  uses 
intended  for  evaluation  results.  This  statement  of  Bureau  objectives  for  evaluation 
and  intended  use  of  evaluation  results  was  requested  by  transit  operators  as  a 
prerequisite  for  their  cooperation. 

This  letter  is  not  meant  to  provide  an  exhaustive  record  of  what  was  said  at  the 
forum.  It  only  cites  some  comments  which,  when  taken  together,  form  a  program 
for  action. 

Opportunity  for  Coop>eration 

At  the  forum  F.  Norman  (Pinky)  Hill,  a  current  ex-officio  Member  of  the  Board  of 
Directors  of  the  American  Public  Transit  Association  and  a  past  president  of  the 
former  American  Transit  Association,  explained  that  the  Bureau's  effort  to 
evaluate  public  transit  services  is  part  of  a  nationwide  trend.  He  pointed  out  that 
this  process  provides  transit  operators  the  opportunity  to  have  a  say  in  the 
development  of  the  methodology.  He  also  pointed  out  that  it  is  in  the  interest  of 
transit  operators  to  use  this  opportunity  and  not  close  the  door  to  it. 

Steering  Committee 

The  vehicle  for  cooperation  between  the  Bureau  and  interested  parties  is  the 
steering  committee  which  has  been  overseeing  the  work  of  Peat,  Marwick,  Mitchell 
&  Co.  The  committee  is  being  enlarged  to  include  two  additional  transit  operators. 
Bob  Foy  (Flint)  and  Terry  Cooper  (Kalamazoo),  and  a  representative  from  UMTA, 
Mike  Higginson. 
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Doug  Gerleman  of  UMTA  suggested  that  the  steering  committee  should  be  charged 
to  answer  the  following  questions: 

What  kind  of  report  is  to  be  submitted  to  the  Legislature? 

What    should    be    done    with    the    methodology    developed    by  the 

consultants? 

Answering  the  above  two  questions  will  be  the  initial  objectives  of  the  steering 
committee. 

Primary  Objectives  of  Doing  Evaluation 

The  Bureau's  primary  objectives  in  developing  and  implementing  an  evaluation 
methodology  are: 

1.  To  comply  with:  (a)  the  Governor's  Budget  Development  Policy;  (b) 
certain  Acts  of  the  Legislature;  and,  (c)  some  recommendations  made 
by  the  Auditor  General.  The  relevant  statements  in  these 
policy/ Acts/recommendations  are  quoted  for  your  information  in  the 
attachment  entitled  "Mandate  to  Evaluate  Bureau  Programs  and 
Projects." 

2.  To  create  a  base  of  data  and  information  for  the  use  of  transit 
operators,  their  local  boards,  local  units  of  government  and  the  State  in 
performing  their  management  roles. 

Subsidiary  Objectives  of  Doing  Evaluation 

The  policy/ Acts/recommendations  mentioned  above  form  a  totality  which 
translates  into  the  following  subsidiary  Bureau  Objectives: 

1.  To  prepare  an  annual  report  on  all  Bureau  programs  and  projects. 

2.  To  provide  (in  the  report)  information  regarding  the  efficiency  and 
effectiveness  (impacts)  of  these  programs  and  projects.  The  report  will 
also  contain  an  account  of  expenditures  from  the  Comprehensive 
Transportation  Fund  and  the  status  of  multiyear  funding  commitments. 
These  accounts  are  prepared  separately. 

Objectives  Specific  to  Transit 

The  above  primary  and  subsidiary  objectives  can  be  defined*  in  more  detail 
according  to  program  and  transportation  mode.  The  objectives  of  conducting 
evaluations  which  ore  specific  to  transit  systems  and  consistent  with  the  primary 
and  subsidiary  objectives  are: 
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1.  To  prepare  an  annual  report  which  will  increase  the  understanding  about 
transit  operations  and  performance  in  the  State  of  Michigan  by  the 
State  Department  of  Transportation,  the  State  Legislature,  the 
Governor's  office,  local  governments  and  other  interests; 

2.  To  monitor  the  efficiency  and  effectiveness  of  the  use  of  public  funds 
for  public  transit  service; 

3.  To  facilitate  the  exchange  of  information  among  transit  systems 
particularly  in  areas  where  there  are  innovative  operations  and 
outstanding  p>erformance;  and 

4.  To  identify  opportunities  to  improve  transit  performance  by  promoting 
more  efficient  and  effective  transit  services. 

Use  of  Evaluation  Results  and  Reports 

The  Bureau  intends  to  use  the  evaluation  results  and  reports  for  the  following 
purposes: 

as  reports  or  basis  of  reports  to  the  Governor,  the  Legislature  and  the 
Auditor  General; 

as  a  base  of  data  and  information  for  program  development  and 
administration  by  transit  management,  local  governments  and  the  state; 

as  publicly  available  reports  documenting  the  efficiency  and 
effectiveness  of  transit  services  and  improvements  achieved; 

as  a  means  of  identifying  exemplary  or  innovative  performance  for 
exchange  of  ideas  among  transit  operators;  and 

as  a  means  of  identifying  any  deficiencies  and  opportunities  to  improve 
transit  service  performance. 

Steering  Committee  Meeting 

We  would  like  to  schedule  a  Steering  Committee  meeting  at  1:30  p.m.  on  Tuesday, 
December  4,  1979  in  the  Fourth  Floor  Conference  Room  of  the  Transportation 
Building.  The  agenda  for  the  meeting  is  to  discuss: 

1.  the  type  of  evaluation  report  to  be  submitted  to  the  Governor,  the 
Legislature,  and  the  Auditor  General;  and 

2.  whether  and  how  the  methodology  that  has  been  developed  would  be 
useful  in  writing  the  report. 

y  This  meeting  was  postponed  until  February,  1980,  when  the  work  plan 
for  the  project  to  Test  and  Refine  the  Evaluation  Methodology  was 
approved. 
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Any  ideas  you  can  provide  to  the  Committee  will  be  appreciated. 

If  there  are  any  questions  regarding  this  letter  and  the  coming  activities,  please 
call  me  at  517/373-2834  or  Angel  Fandialan,  Chairman  of  the  Steering  Committee, 


at  517/373-6572. 


Sincerely, 


Maxie  C.  JacksOTi,  Jr. 
Assistant  Deputy  Director 
Urban  and  Public  Transportation 


Attachments: 


t.      List  of  Forum  Participants 
2.      Mandate  to  Evaluate  Bureau 
Programs  and  Projects 
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Participants  at  the  Forum 
on  a  Proposed  Methodology  for  Transit  Evaluation 
October  30,  1979 


U.  S.  Government 

UMTA,  Washington,  D.C.,  Office  of  Transportation  Management  -  Brian  Cudahy, 
Don  Chapman 

UMTA,  Chicago,  Planning  Division  -  Douglas  Gerleman,  Mike  Higginson 

Michigan  Mid-size  Urban  Transit  Systems 

Ann  Arbor  Transportation  Authority  -  Richard  Simonetta,  Executive  Director 
Battle  Creek  Transit  -  Wayne  Wiley,  Acting  Transit  Manager 
Bay  County  Metropolitan  Transportation  Authority  -  Lewis  J.  Gordon,  General 
Manager 

Flint  Mass  Transportation  Authority  -  Robert  Foy 

Grand  Rapids  Area  Transit  Authority  -  Don  Edmondson,  General  Manager 
Jackson  Transit  System  -  Carl  S.  Buchanan,  General  Manager 
Kalamazoo  Department  of  Transportation  -  Terry  Cooper,  Acting  Transit 
Division  Head 

Lansing,  Capital  Area  Transportation  Authority  -  Clare  Loudenslager,  Executive 
Director 

Muskegon  Area  Transit  System  -  Ernie  Palmer,  Transit  Systems  Supervisor 
Saginaw  Transit  Service  -  Mark  J.  Dorfman,  Public  Transit  Administrator 

State  Government 

Sen.  Plawecki's  staff  -  Walter  Heinritzi 
Rep.  Ryan's  staff  -  Carol  Norris 
Sen.  DeSana's  staff  -  Patrick  Harrington 
House  Fiscal  Agency  -  Phil  Kazmierski 
Senate  Fiscal  Agency  -  Jesse  Brown 

Auditor  General  -  Frank  Baciga!,  Assistant  Auditor  General 
Transportation  Commission  Audit  -  James  McMahon,  Chief  Examiner 
Bureau  of  Urban  and  Public  Transportation  -  Maxie  Jackson,  Jr.,  Gary  Barrett, 
William  Beachler 

Methodology  Development  Steering  Committee 

Angel  Fandialan,  Chairman  -  Bureau  of  Urban  and  Public  TransF>ortation 
Richard  Beattie  -  Southeastern  Michigan  Transportation  Authority 
Terry  Cooper  -  Kalamazoo  Department  of  Transportation 
Frank  DeRose  -  UPTRAN,  Bus  Transit  Division 
James  Dunn  -  Michigan  Public  Transit  Association 

Michael  Eberlein  -  Office  of  Inter-Agency  Transportation  Coordination,  MDOT 

Robert  Foy  -  Flint  Mass  Transportation  Authority 

Mike  Higginson  -  Urban  Mass  Transportation  Administration 

Keith  Killough  -  Southeastern  Michigan  Council  of  Governments 

Kunwor  Rajendra  -  Lansing  Planning  Department 
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James  Roach  -  Mass  Transportation  Planning  Section,  MDOT 
Gordon  Szlachetka  -  Tri-County  Regional  Planning  Commission 
David  Youngs  -  Michigan  Department  of  Management  and  Budget 

Peat,  Marwick,  Mitchell  &  Co.  -  John  Bennett,  Jim  Holec,  Diane  Schwoger 

Others 

F.  Norman  Hill,  Ex-Officio  Member,  Board  of  Directors,  American  Public 
Transit  Association 
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Mandate  to  Evaluate  Bureau  Programs  and  Projects 


The  mandate  to  develop  evaluation  methodologies,  to  perform  evaluations  and  to 
report  evaluation  findings  originates  from: 

A.     The  Governor 

The  following  general  concepts  are  contained  in  the  Governor's  Budget 
Development  Policy  for  1980-81.  They  provide  a  framework  upon  which 
the  1980-81  Departmental  Management  Plan  is  predicated. 

1.  Fiscal  Review  and  Constraints 

"Sound  budgetary  development  continues  to  require  that  State 
agencies  submit  resource  and  performance  information  which  is 
demonstrably  related  to  program  priorities  and  to  provable 
efficiency  and  effectiveness.  In  cooperation  with  Department  of 
Management  and  Budget  staff,  agencies  are  expected  to  monitor 
and  evaluate  programs  still  more  intensively."  (quote  excludes 
underlines) 

2.  Program  Review 

"In  this  era  of  growing  demands  for  accountability,  I  continue  to 
be  concerned  with  improving  the  quality  of  agency  management 
plans  in  order  to  enhance  the  analytic  basis  for  the  budget 
decisions  that  I  must  make.  I  am  convinced  more  than  ever  that 
program  needs,  impacts  and  efficiencies  should  be  measured  and 
displayed  in  ways  which  allow  the  Legislature  and  the  citizens  of 
Michigan  to  judge  in  an  enlightened  manner  the  performance  of 
state  government. 

Program  information  submitted  in  the  management  plans— 
need/demand,  outputs,  and  impacts— continually  should  be 
upgraded  and  used  as  the  data  base  of  program  budget  aholysis, 
evaluations,  requests,  and  recommendations.  Budget  staff  will 
continue  to  work  with  you  and  with  legislative  staff  to  improve 
the  usefulness  of  such  information." 

B.      The  Legislature 

I.      Appropriation  Bills 

Act  No.  468  of  1978  and  Act  No.  108  of  1979  contain  the  following 
mandate  in  different  paragraphs  but  in  identical  wordings: 

"(I)  Each  department  or  agency  shall  prepare  and  furnish  to 
the  department  of  management  and  budget,  the  senate  and  house 
appropriations  committees,  and  the  senate  and  house  fiscal 
agencies,  a  statement  establishing  the  key  resource  and 
performance  data  which  will  be  used  during  implementation  to 
demonstrate  the  efficiency  and  effectiveness  of  each  program  or 
project. 
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(2)  When  it  appears  to  a  program  manager  or  department 
director  that  a  program  will  not  meet  the  program  performance 
commitments  as  submitted  to  the  legislature,  the  program 
manager  or  department  director  shall  submit  a  written  report  to 
the  members  of  the  senate  and  house  appropriations  committees. 
The  report  shall  include  a  detailed  explanation  as  to  why  the 
program  will  not  meet  its  anticipated  level  of  productivity  and 
justification  as  to  why  the  commitments  cannot  be  met.  As  used 
in  this  subsection,  "program  manager"  means  the  individual 
responsible  for  the  Implementation  or  the  ongoing  management  of 
a  program." 

2.      Act  No.  51,  Section  IOh(l)(b) 

Sec.  lOh(l)  states  that  "by  April  I  of  each  year  the  state 
transportation  commission  shall  report  to  each  member  of  the 
legislature,  the  governor,  and  the  auditor  general  its  recom- 
mendations for  a  transportation  program.  The  report  shall  specify 
the  following:.  .  . 

(b)  An  account  of  all  expenditures  of  funds  distributed  from 
the  state  trunkline  fund  and  the  comprehensive  transportation 
fund  to  the  department  of  transportation,  eligible  authorities,  and 
eligible  governmental  agencies,  and  the  progress  made  by  the 
department  of  transportation,  eligible  authorities,  and  eligible 
governmental  agencies  in  carrying  out  the  approved  transportation 
programs  in  the  preceding  fiscal  year  through  the  use  of  those 
funds.  The  progress  report  shall  be  made  based  on  information 
supplied  to  the  state  transportation  commission  on  forms 
authorized  by  the  federal  department  of  transportation.  For  those 
eligible  authorities  and  eligible  governmental  agencies  not 
receiving  federal  funds  pursuant  to  the  urban  mass  transportation 
act  of  1964,  49  U.S.C.  1601  to  1614,  the  progress  report  shall  be 
made  upon  forms  supplied  by  the  department  of  transportation. 
The  progress  report  shall  also  contain  the  whole  amount  of  the 
expenses  of  the  department  of  transportation  for  the  fiscal  year." 

The  Auditor  General 

The  Audit  Report,  May  24,  1979,  pages  4  and  5  contains  the  following: 

"Evaluating  Improvements  and  Efficiency  of  Bus  Transportation 
Services 

I.  The  bureau  has  developed  a  quarterly  report  which  provides 
information  on  State-supported  public  transportation  services. 
The  report  shows  comparisons  between  the  current  reporting 
period  and  the  some  period  of  the  prior  year.  However,  the 
bureau  has  not  developed  criteria  and  methods  to  evaluate  the 
performance  and  effectiveness  of  bus  transportation  systems  .  .  . 
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WE  AGAIN  RECOMMEND  THE  DEPARTMENT: 

(a)  DEVELOP  CRITERIA  AND  METHODS  TO  DETERMINE  IF 
STATE  GAS  TAXES  ARE  USED  TO  OPERATE  EFFICIENT 
AND  EFFECTIVE  PUBLIC  TRANSPORTATION  SYSTEMS. 

(b)  IN  COOPERATION  WITH  TRANSIT  AGENCIES  AND 
AUTHORITIES,  DEVELOP  UNIFORM  POLICIES  AND 
METHODS  TO  MANAGE  AND  CONTROL  THE  OPERATING 
COSTS  OF  TRANSIT  SYSTEMS  AND  ROUTES." 
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DRAFT — Subject 
to  Revision 

UPTRAN's  STATEMENT  ON  TRANSPORTATION  SYSTEM  EVALUATION 


Act  51  of  the  State  of  Michigan  Public  Acts  of  1951,  as 
amended,  provides  for  financial  assistance  to  eligible 
governmental  authorities  and  eligible  governmental  agencies 
involved  in  the  delivery,   improvement  and  maintenance  of 
transportation  services.     The  purpose  of  using  State  funds 
for  public  transportation  assistance  is  to  provide  and 
enhance  transportation  services  in  Michigan. 

As  the  administrator  of  funds  authorized  under  Act  51, 
it  is  the  position  of  the  Department  of  Transportation  that 
it  is  responsible  to: 

1.  Actively  participate  in  ensuring  that  these  funds 
are  used  to  provide  efficient  and  effective  trans- 
portation services; 

2.  Monitor  'the  distribution  and  use  of  public  funds 
for  transportation  to  ensure  fiscal  responsibility 
in  the  use  of  limited  resources;  and 

3.  Assist  local  areas  to  develop  and  improve  transpor- 
tation services  with  minimal  administrative  and 
regulatory  obstruction. 


To  fulfill  this  responsibility,  the  Department  of 
Transportation  intends  to  undertake  a  program  of  ongoing  and 
routine  evaluation  of  the  transportation  systems  funded 
with  State  revenues.     This  program  has  been  initiated  for 
the  development  of  a  Public  Transportation  System  Evaluation 
Methodology  for  the  mid-size  transit  systems,   i.e.,  those 
in  Michigan's  urbanized  areas  except  SEMTA. 

This  methodology  is  scheduled  for  test  application 
during  the  current  fiscal  year.     Once  refined  and  modified 
to  reflect  the  findings  of  the  test  application  and  the 
inputs  of  mid-size  public  transportation  operators,  the 
methodology  will  serve  as  the  model  upon  which  subsequent 
evaluation  efforts  of  other  transportation  modes  will  be 
based . 

By  highlighting  the  principles  guiding  the  State  in 
this  effort,  this  statement  clarifies  objectives  for  and  the 
intended  use  of  the  results  from  this  evaluation  process. 
This  statement  also  describes  the  main  features  of  the 
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Public  Transportation  System  Evaluation  Methodology  devel- 
oped by  the  State  and  shows  the  relationship  between  these 
features  and  the  principles  guiding  the  State's  overall 
efforts  in  this  area. 

PRINCIPLES  GUIDING  THE  DEVELOPMENT  OF  TRANSPORTATION  SYSTEM 
EVALUATION  METHODOLOGIES  FOR  STATEWIDE  APPLICATION 

The  State  of  Michigan  intends  to  develop  a  routine  and 
ongoing  evaluation  process  for  each  transportation  program 
which  receives  State  financial  assistance.     The  principles 
guiding  the  State  in  this  effort  include: 


1.  The  State  has  the  responsibility  to  undertake 
transportation  system  evaluations  to  ensure  the 
efficient,  effective  and  economical  use  of  public 
funds  used  to  support  transportation  services; 

2.  The  State's  overall  objective  for  transportation 
system  evaluation  is  to  contribute  to  the  improve- 
ment and  development  of  transportation  services 
throughout  Michigan.     To  achieve  this  overall 
objective,  the  State  intends  to  use  the  results  of 
transportation  system  evaluations  for  the  following 
purposes : 

o    To  facilitate  the  identification  and  exchange  of 
unique  and  creative  management  and  operating 
practices  throughout  the  State; 

o    To  provide  a  framework  for  self -evaluation 
activities  by  local  transportation  providers; 

o    To  identify  opportunities  for  improving  manage- 
ment and  operating  practices  and  cooperatively 
formulate  actions  designed  to  take  advantage  of 
these  opportunities;  and 

o    To  strengthen  the  local  programming  and  budget- 
ing process  by  integrating  the  evaluation 
methodology  and  results  in  the  development  and 
review  of  local  transportation  plans. 

3.  The  State  recognizes  the  need  to  conduct  transpor- 
tation system  evaluations  within  a  framework  that 
fosters  cooperation  witlj  local  transportation 
service  providers  and  at  the  same  time  encourages 
the  continued  independence  of  these  providers  to 
creatively  improve  and  enhance  transportation 
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services  in  response  to  specific  local  needs  and 
desires.     In  this  spirit,  the  State  is  committed  to 
the  development  of  evaluation  methodologies  that 
both  serve  the  needs  of  the  State  (that  is,  satis- 
fy the  responsibilities  of  the  State),  and  at  the 
same  time  offer  direct  benefits  to  individual 
transportation  system  providers  and  will  not 
infringe  on  the  role  of  these  providers  as  the 
day-to-day  managers  of  transportation  resources. 

4.     The  State  has  the  responsibilit iy  to  minimize  the 
use  of  limited  public  revenues  for  administrative 
activities  and  is,  therefore,  committed  to  the 
development  of  evaluation  methodologies  that  (a) 
maximize  the  use  of  existing  data  and  information 
(b)  are  not  redundant  with  existing  evaluation 
activities  conducted  by  the  State  or  by  the  local 
providers,  diverting  their  attention  from  their 
daily  resp  isibilities. 


PUBLIC  TRANSPORTATION  SYSTEM  EVALUATION  METHODOLOGY 

The  State  will  use  the  principles  outlined  above  in  the 
development  and  ongoing  refinement  of  an  evaluation  method- 
ology for  mid-size  transit  systems  in  Michigan.  The 
methodology  was  designed  to  produce  information  intended 
to : 

o     facilitate  the  identification  and  exchange  of  unique 
and  creative  transit  management  and  operating 
practices  throughout  the  State; 

o    identify  opportunities  for  improving  transit  manage- 
ment and  operating  practices; 

o    demonstrate  the  benefits  achieved  through  investment 
in  public  transportation; 

o    define  actions  designed  to  improve  the  efficient  and 
effective  use  of  State  funds  devoted  to  public 
transportation  assistance;  and 

o     increase  understanding  about  transit  operations  and 
performance  in  the  State  of  Michigan  by  the  State 
Department  of  Transportation,  the  State  Legislature, 
and  other  interests. 
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The  methodology  avoids  placing  an  excessive  reporting 
burden  on  local  transit  operators,  by  relying  largely  on 
federal  Section  15  data  routinely  reported  by  the  operators 
each  fiscal  year. 

The  State  has  embarked  on  a  program  of  transition  from 
the  development  sized  transit  operators  in  Michigan.  This 
transition  program  provides  for: 

o     further  refinement  and  tailoring  of  the  evaluation 
methodology  to  better  meet  the  needs  of  the  State 
and  mid-size  transit  operators; 

o    statewide  pilot  testing  of  an  evaluation  methodology 
in  all  mid-size  transit  systems  in  Michigan;  and 

o    preparation  of  the  format  and  content  of  future 
annual  progress  reports  to  the  Michigan  State 
Legislature. 


To  complement  this  effort,  the  State  has  also  agreed  to 
prepare  an  evaluation  manual  for  the  use  and  benefit  of  the 
raid-size  transit  operators  in  conducting  self -evaluat ion . 

The  State  believes  that  the  period  of  statewide  pilot 
testing  in  particular  will  afford  all  of  the  mid-size 
transit  operators  in  Michigan  the  opportunity  to  be  directly 
involved  in  applying  the  methodology  both  for  their  own  use 
in  self-evaluation  and  for  the  State  in  gathering  informa- 
tion on  transit  performance.     Through  this  testing  phase, 
the  State  intends  to  work  closely  with  the  mid-size  transit 
operators  to  ensure  proper  adherence  to  the  principles 
guiding  the  State's  development  efforts  in  this  area. 
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